Year One Evaluation Report/Impact Study: 
Illinois Striving Readers 


March 26, 2012 


AG 


RESEARCH 


Year One Evaluation Report/Impact Study: 
Illinois Striving Readers 


Presented to: 

Shannon Mitchell, Education Program Specialist 
Office of Elementary & Secondary Education 
Stefanie R. Schmidt, Project Officer 

Institute of Education Sciences 

U.S. Department of Education 

400 Maryland Avenue, SW; Room 3E336 
Washington, DC 20202 


Submitted by: 

RMC Research Corporation 

1501 Wilson Boulevard, Suite 1250 
Arlington, VA 22209 

Phone: 703.558.4000 

Fax: 703.558.4823 


Authors 

Dimiter Dimitrov, Principal Investigator 
Sonia Jurich, Project Director 

Michael Frye 

Jill Lammert 

Sarah Sayko 

Laura Taylor 


March 26, 2012 


TABLE OF CONTENTS 


"FABLE OB CONUENYES a sssisscicieihossivatoseseevcsaudusseakivsdetessenepodconsveousniosbendeniessusnanisenttbasonxevassoenssniues i 
BGES. 2 OR PAB LES qccaascosdssnsverssshsvies ccsnstawasnncvsedadyaxeeasybisenssonaetassopeoussee deekacdnsevseacsnss duces veccespeseacnnns iii 
ISIS TOP FIGURES oi seccccusencsestvedsecosscuccscuacdeenssivedsensaov asuiicudisuseususscuouruedeusecsdstededoseusenusseniucuseorsecss iv 
EXE CUTEY E SUMMARY <x ,ccccdssiscconsctscecestspsasvevaustutessaienscobsaesascensussteucessdscSesssnadeaatvassonddossonsenesss v 
INTRODUCTION vacsieicainis css cous wsnstascseatesseus ca cendivel ec tivssstenstewwseuatahecsiesestoactapvessuctuovsductnssedessecetabenne 1 
PART I: INTERVENTION AND LOGIC MODEL .............scsscsssesessscecscccseescessescssesssesessseesees 4 
Description of the intervention model cise tose lotic gidind accu dius plaeicdanetinnl: 4 
Classroom Intervie non esis sated Fale hase 80 AAs Sods hat cases hide caudate ae ie Send aS hf taa 4 

WSG:OF TECHMGIOS Yo. die, gucci eutavaiaiss iuandagessassdyosseusenbnsacdayy sonuse songassdobescaeasensa aad Peameeguaeeceeonoscueel 6 
Professional Ievelop ment ts2ese5 ester ietet testis erst se cestst lease da ret 1h Seah oat oes Nina eda tte reas | 
PASSCOSSITIOIUS czasced bite nacht ss as ts aD ba cs ere aga takes alas steal baedk uses staid aa ee eeatins 8 

Par Ge MpOpUl ali OMs...12,2ot ores veqeaacideeyactans vases belen dees tgceansuae been te uecuaartabeddeas tadeeaaedsedachenavcartehasnane’ 9 
Desired: characteristics. of the teachers ac..cuinicevei iets ce SAAD a te ee 9 
Desired characteristics: Of the: Classrooms ws cco: ct.sacigsleclp nee ates teil eek ees. es 9 
Recommended intensity for the student #..<.2.202: 4.222600. c2cseacsedactesteectecsteeeee edie ndannceeces 9 
Illinois Striving Readers (ISR) Logic Model... eee eeseescesceeeneecsseceeeeeeeeeseecsaecsseesseeeeaeesaeenes 10 
CVE V NCW cats sas cde acts as acme ted a teen ey nen anti Ube ih eal tee a cuales 10 

ISKa professional development Model 24st aa tieeentrien ieee 14 
Planned: classroom 1nStructvOny ss cesccticbdsecc reteset itdocslpoucdedd, sthaeiazeesdaresibietel eaahdietieesamecas 1S 
Peachiers*CHAraCleriS HGS sc:)..t27c24 odes iyscsiet tant ee ra dade nibh gue ti otaeet webade deeb us achuaceanatanidy 16 
ClassroomCharacteri Stas ssy2i5:cis25sistay cee, octheeehsctentes hes) teers ere ete desd ned cacy cede 16 

PRS SO SS UN OTIS oles cl Ae rea Fe ot ert ue a acl ta cess aida NP set Ri naeth NS neh PR ae rete cel 16 

TSR. CH eible SHIGENIS 5 ioiss ccc sigiaeas Wess adageedants sigs dastaayspaneeee Geese Seed ae ane L7 
Expected: Stu demit: QULC ORES sccschadat pete ait hss secede hates weer ati a heddveedet uaa Scat lat 18 
Planning: Year (2009-2010) .v.nta ein Genc au nein ea eee ia 19 
PART Ti: IMPLEMENTATION STUDY osscssessssscssessescossoossssnesessovvevsecsonssesssneoseosonetoessossensecsss 20 
SUA: DESI S052. cc5s 5 is scess vedevaneselcagesseceis opadadaats yates avssas sgesasaaaas gasadeaepaganes sbesdaageestieasspacaaae eyeateas econ 20 
CVT VIC Weber scene tves tate Ns ead ck la ni aiaa tender cua de tee eala cate atthe alan Mla Meliss Mtoe 20 
Research Questions. secu seciteeetalele ogee sc uietutes tena iah 2s hed SU ar eee) eee 20 

Data ol leet) ates seas Ae tie cot eer es Ae nent ken ett ah ur hth At asta ee nc Ait ial ven ea ioscan 21 
Defining. fidelity of implementation sijcctssceccecesdedigesetet eeensscdnives dhnueeeiisetes alan edavtied Ze 
Implémetitatiom Year U (2010-2000) accuse csescaehachissa th cacareedstonte tenet Dat aa a a dackedsh atten Seactl aes 24 
Context of. KPJ mmplementatons. 5:25. csssdie eek ts iGo ats eM ah ee 24 
Professional development model implementation... eee eesceeeeeereeceeeceeeeeseeeeaeeeaeens 2 
Implementation of the classroom MOdel 00... eee eeseeeeceeseecneeceeceeeeeeaeecsaecnseeeeeeesaeeenaeens vA 
Factors influencing the implementation .252.0jccii ot code vesgdedpvestedscteueeetdeveteepl eetade seen odactlecciocdstennacs 33 


Illinois Striving Readers, Implementation Year One Report 


General factors! ss iscsckc tied sich dass sh odedsbedbanacaes db oeteesdh ae sbeedoeds etoadademboods visidead detects s aE 34 


SPECINIC TACLOTS 5-7 ohco vesaniavsletatey stdelvauetedahenes nanigedaSsautes agnuaealtndehen eae eyadesenatoleass aeaesterspannaantacrs 34 
Factors influencing the impact analysis ............ceecccessseceessecesseeeceseeecesceececeeceeeeseneecseeeecseaeeeeaees 37 
PART TE IMIPA CIS PUD YV wassiccessionsinecsnscasveasnansscsnspecssnscavtacinevexsennasessconsnensenscan¥essnetenecshesekecins 38 
Study Designyisisss cscccieveepeanastiesleeiad easiest Wesei MeE ad ea eee 38 
Saimiple SCLC GN x scrogaaindesareatarnssmeacasiucese teteetepaa desea aeloastoneauieedetdies avaaeaaeoeatoamatachnesten 38 

Data COME Cin sess ssccesayees cies shanicaes praceageaizecaash wealeivisesdieasennsagh pas dana ease are eeu 43 

Data AMal y S18 tice sto: cep ounce wade wae Sieve ehey acs gsaanaey AP ewan guns tes ew tatoos taken BOS 44 

Impact on Students at the End of Year One... ee eeeceseesseecneeceeceeeeesaeecaeceeeesseeesaeesaeenee 45 
Impact ‘on-student reading Promciemey sé. jis chev ee ss cgceadsdesnasesengcdadetcbes wats ta cecusshensaretoatederaonass 45 
ACdiOmal Atal VSES ad, y.cctees cs sicevsnsaceeuhh,vaben cov aadevasucasgubava aenosanceasaseead bes aeaayenaceetne mteacavactoeapncens 48 
PSX PO RIMC NFA, AMA YS OS $x c6 tests nee te 8 tt en ee te hs Ae A te Br CI, 48 
CONCLUSION scsasicssvusiecnassscsseseucsccesnicdsonssedeonssvnssodsownsecnasusesesssuedssgsumeness weeuessasenan ccssundosnpsus easesats 50 
BRE HEREC NCES cssischsas cs astacevssoguacedssontnonsebeausuGenasssavinstedecesscsnsdestureigcensuepeennctenosseasenscnsbegssebncsnisoeneans 52 
APPENDIX A: Summary Statistics for Outcome Measul[ ...........sccscccsssscccsssccsesscsssssceseeces 55 
APPENDIX B: Implementation study protocols ..............ccsscccsssscssssscssssccssscccssscssssescsssscceseaces 56 


Illinois Striving Readers, Implementation Year One Report 


LIST OF TABLES 


Table 1. Professional development activities related to the Illinois Striving Readers (summer 


ZU SPOT DOV Lys sed alee a i ea eS fe aa loses ae a a 14 
Table 2. Illinois Striving Readers — List of assessment ..........eeceesceeeseceseceeeeeeneecsaeeneeeeaeeesaeens 17 
Table 3. Illinois Striving Readers schools (SY 2008-2009) ..........eesceseesseeeeseecsseeeseesseeesaeesaeens 18 
Table 4. Implementation evaluation — data sources and data collection processes .............06+5 Zn 
Table 5. Alignment between IFI and evaluator observation rubIic ..0.... ee eee esse esteceeeeeeeeeeeeens 23 
Table 6. Calculating the classroom implementation fidelity SCOT... eee eeeesteceeeeeeeeeeeeees 24 
Table 7. Student Enrollment in Passport Reading Journeys U1... eee eeeeeeeeeeeeeeneeeeneeeeeaes 25 
Fables. Hours of professional developiient 655.3 ceevecsscte vavesticus te ends save nees chee cesses Raede Gewese 26 
Table-9. Hours of individualized supports :ic.c05.206Ga cence a ain i ease 26 
Table 10. Index of fidelity of implementation — professional development model................... ay. 
Table 11. Lessons completed during class time... eee eeeceesceseeeeseeceaeceseeeseeeeacecsaeeneesseeeeaeees 28 
Table 12. School closures and class cancellations ............eseeeeeseecseceseceseeeeseeeeaeecsaeeseeesaeesnaeens 28 
Table 13. Student reported: attendances. pcs cclaaccicaieicon sec airdeeinn de aie 29 
Table (42Pacing Of instru t On ss soca ewcdscct aa tee Gadel Ge aeutnd a ead eee eed oe elena 29 
Table 15. Example of reporting on the benchmark assessments ............ceecceeeeeeseeeseceeeeeeeeeeaeens 31 
Table 16: Scoring for the classroom Observation rUbTIC ........ eee eee eeeesneceeeeeseeesacecaecnseeeneessaeens 32 
Table 7: Fidelity-scores per School 4.2.0). chai G Rae eee Ree nas 33 
Table 18. 1T Tl breakdowns by’school 9.5.21. enciceg eateries i yeetiv wed Meee ee 40 
Table 19. Demographic breakdown of ITT group... cee eee eeeeeeeseecneeceeeeeeeeeeaeecaesnseenseeeeneees 40 
Table 20. List of independent variables used with the experimental design model .................. 43 
Table 21. A two-level Hierarchical Linear analysis under the model defined by equations 1 

MINN, 5d cusiiec sles Poeievsn unas laden ea vay cae cvs tna cag edad dues Din Sede thea oasie lea dade odenw ld loan Was eda 45 
Table 22. OSL regression for the prediction of posttest scores on the Grade 9 GMRT®.......... 46 
Table 23. Two-level Hierarchical Linear analysis under the model defined in equations | and 

2, with the Grade 9 EXPLORE® scores as the outcome variable ........ccccscscssssseseeseseeeeseseeees 47 
Table 24. OSL regression for the prediction of posttest scores on the Grade 9 EXPLORE. ..... 48 
Table 25. Adjusted means and their standard deviations on the posttest Grade 9 GMRT® by 

(PEeALMeNE CONMIION ANG BONCE Mt uelestcsentockley tics aan deateianes cid cunt tay nae uN Gate Nena aati 48 
Table 26. Adjusted means and standard deviations on the posttest Grade 9 EXPLORE® by 

treatment condition and ethnicity ...c..cccccscsssssssssssssssssssssssesssssssssssssssssssssssesssesssssssssessssees 49 


Illinois Striving Readers, Implementation Year One Report 


iii 


LIST OF FIGURES 


Figure 1. Logic Model for the study of the Illinois Striving Readers... ee eeeeeeeeeeeeeeneeees 11 
Figure 2. Illinois Striving Readers —Implementation and study interaction ......... eee eeeeeeeee 13 
Figure 3. Consort diagram — ITT group attrition in regards to EXPLORE® .........csscessesssesseeoes 4] 


Figure 4. Consort diagram — ITT group attrition in regards to GMRT® 


Illinois Striving Readers, Implementation Year One Report 


EXECUTIVE SUMMARY 


Illinois State Board of Education (ISBE) competed for and was awarded a four-year grant 
Striving Readers program from the U.S. Department of Education (USED). The Illinois 
Striving Readers (ISR) Project had two purposes: (a) implement a supplemental reading 
intervention for students in ninth grade who were reading below grade level, and (b) study the 
impact of the intervention on students’ performance on standardized assessments using a 
randomized control trial design. 


ISBE selected Passport Reading Journeys III (PRJ IIT) as the supplemental reading 
intervention to be implemented. PRJ III, published by Cambium Learning Group, is a highly 
structured reading program that is organized in 14 Expedition lesson series taught daily in 50- 
minute periods during one school year. Each Expedition comprises a total of 10 lessons that 
alternate between whole group and individual practice. Lessons 5 and 10 are dedicated to 
reviewing, expanding, and assessing. During these two lessons, the teachers can choose a 
number of strategies in a menu of options offered by the publisher. Depending on the number 
of options that the teachers decide to use, Lessons 5 or 10 may take two to three days. The 
materials for the intervention include teacher guidelines, student workbooks, DVDs, and a 
library of fiction and non-fiction books and magazines that are age-appropriate and have the 
purpose of engaging the adolescent reader. 


ISR involved six high schools in four school districts across Illinois. All schools were Title I- 
eligible schools that had not made, or were at-risk of not making, adequate yearly progress 
requirements under the No Child Left Behind Act of 2001. Students who scored at the two 
lowest quartiles on grade 8 EXPLORE ® were eligible for the study. Two exclusion criteria 
were proposed: (a) students with Individualized Education Plans (IEP) that precluded their 
participation in the study, and (b) students whose parents requested that their children not 
participate in the study. The evaluators discussed with each school district the requirements of 
the Federal Policy for Protection of Human Subjects (34 CFR, Part 97). Recommendations 
were made for the schools to follow their procedures for parental notification using an “opt- 
out” (rather than “opt-in’’) option. 


After the exclusion criteria were considered, eligible students were randomly assigned to 
treatment and control groups. Students in the treatment group were instructed using PRJ III, 
while students in the control group enrolled in elective classes that did not provide 
supplemental reading instruction. A total of 855 students participated in the study. The project 
had two outcomes: (a) at least 50 percent of the treatment group students would demonstrate a 
gain in reading achievement, at minimum, of one grade level or its equivalent (objective 1.3); 
and (b) 75 percent of the students in the treatment cohorts would score at or above proficient 
on the state reading assessment (objective 1.4). 


Each school hired a Reading Intervention Teacher (RIT) who was expected to have 80 percent 
or more of his or her time dedicated to teaching PRJ I/I. All RITs were required to have a 
valid Illinois teaching license with a reading endorsement, two to three years of teaching 
experience, an understanding of the Response to Intervention process, and proven classroom 
management skills. As part of the contract, the RITs were required to attend all of the 
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professional development activities related to the intervention and to implement PRJ IT with 
fidelity. Each RIT was to have three to four classes of no more than 25 students each. 
Cambium Learning Group was responsible for providing professional development and 
supports to the RITs. 


RMC Research Corporation (RMC) competed for and was awarded the contract to conduct the 
ISR evaluation. The evaluation comprised an implementation and an impact study. The 
implementation study answered the question of whether the intervention was implemented with 
fidelity to the original model. To answer this question, the evaluators conducted monthly 
phone calls with the RITs and yearly interviews with the project coordinator, local education 
agencies (LEA) liaisons, and Voyager representatives. Additionally, the evaluators observed 
PRJ III classrooms. The impact study focused on the effects of the intervention on student 
academic performance, as measured by results in the Grade 9 Gates-MacGinitie Reading Tests 
(GMRT) 4" edition and Grade 9 EXPLORE. ® 


ISR was planned as a four year study with the first year dedicated to planning followed by 
three years of implementation. The project would serve three cohorts of grade 9 students 
(2010-2011, 2011-2012, and 2012-2013). In April 2010, the USED announced that the 
Congress had cut funding for the Striving Readers program and the project was to conclude by 
the end of the first implementation year. This report describes and discusses preliminary 
findings from this first year of implementation and the one-year impact of the program on 
treatment students compared to control students. A summary of key findings are presented 
next. 


Implementation Study 


e Findings from the first implementation year suggest that all but one school attained 
adequate implementation of the professional development model, while the fidelity of 
the classroom model ranged from inadequate to high fidelity. 


e Two factors were seen as influencing the fidelity of implementation across the six 
schools: professional development and supports, and program format, particularly 
pacing and routine. 


e Specific factors that were seen as influencing the schools differently included: 
classroom management, resulting mostly from the inclusion of students who fit the 
eligibility definition but resented the intervention; classroom space, which had an 
impact on the teachers’ ability to organize group instruction and manage student 
behavior; actual time in instruction; and familiarity with the program (or time in 
implementation). 


e Although teachers in all six schools had similar backgrounds (in terms of licensing and 
reading endorsement), attended similar intervention-related professional development, 
and received individualized supports, their ability to deal effectively with the factors 
above varied across schools. Data from the first implementation year suggest that the 
“specific factors” had moderator roles, either minimizing or exacerbating the impact of 
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the general factors on the fidelity of implementation. This hypothesis could not be 
tested due to the abrupt ending of the study. 


Impact Study 


e Of the 1,985 students across the six high schools a total of 855 students were eligible for 
the study. These students were randomly assigned to treatment (427 students) and control 
(428 students) groups. 


e Outcome analysis results on both reading measures — the grade 9 ACT EXPLORE® 
Reading and the grade 9 GMRT® — showed no significant impact of the reading 
intervention on students’ reading achievement. 


o Estimates of between-school variance of the PRJ III effects were not 
statistically significant thus indicating that the impact of PRJ III did not vary 
across schools. 


o Student inclusion in different demographic (gender, ethnicity, special education, 


free and reduced lunch, and English language proficiency) subgroups yielded no 
or negligible differences on the reading outcome measures. 
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INTRODUCTION 


There is a collective effort underway in America to ready students for college and career after 
high school graduation (President’s Council of Advisors on Science and Technology, 2010). 
To accomplish this goal, students must be adequately prepared to meet the increased literacy 
demands they will face in middle and high school. They also must have the necessary supports 
and strategies in place for overcoming learning barriers to progress in their educational path. 
In reality, many adolescent readers struggle to master the most basic of literacy skills. 
According to the National Center for Education Statistics (NCES), more than 8 million 
students in grades 4-12 are struggling readers (USED, 2003). Data from the National 
Assessment of Educational Progress (NAEP) show that results from the NAEP reading 
assessment have remained unchanged over time; in 2011, only 34 percent of eighth graders 
scored at or above proficiency (NCES, 2011). Research indicates that high school students in 
the lowest 25 percent of their class are 20 times more likely to drop out than their highest 
performing peers (Carnevale, 2001). Additionally, of those who graduate, approximately 25 
percent enroll in literacy remedial courses in postsecondary education (USED, 2003). 


The most serious challenges faced by struggling adolescent readers are difficulties with 
decoding, slow and labored reading, lack of background knowledge, and limited vocabulary 
(Beck, McKeown, & Kucan, 2002). Many of the poor and minority students who currently 
perform below the 30th percentile in reading skills entered school with academic vocabularies 
already only half the size of their middle-class counterparts (Beck, McKeown, & Kucan, 
2002). Additionally, they have fewer opportunities to acquire the kinds of active reading 
comprehension strategies that become increasingly important as text increases in size and 
complexity after third grade (Torgesen, 2005). As a result, these deficits interfere with 
struggling adolescent readers’ ability to develop higher level literacy skills and cause cognitive 
challenges that may become impossible to overcome. 


Research indicates that a crucial component to addressing struggling adolescent readers’ needs 
is improving literacy instruction and intervention practices in schools. This includes providing 
explicit vocabulary instruction, direct and explicit comprehension strategy instruction, 
opportunities for extended discussion of text meaning and interpretation, and increasing 
student motivation and engagement in literacy learning (Kamil, Borman, Dole, Kral, Salinger, 
& Torgesen, 2008). In addition, intensive and individualized interventions for those who 
struggle most must be available and delivered by trained specialists (Kamil et al., 2008). These 
interventions should target students’ instructional needs, occur in small group settings during 
extended learning periods, happen with increased frequency, and incorporate opportunities to 
monitor student progress. Alternatively, research suggests that it is important to avoid 
intervention programs that have insufficient intensity, weak instruction in word study skills, 
and little or no direct instruction in comprehension strategies, as they have limited 
effectiveness (Scammacca, Roberts, Vaughn, Edmonds, Wexler, Reutebuch, & Torgesen, 
2007). Because interventions for students’ still mastering basic reading skills tend to stabilize 
rather than remediate the relative reading deficiency (Scammacca et al., 2007), careful 
selection of research-based adolescent reading programs is essential. 
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Addressing research findings regarding the needs of adolescent who are struggling readers, the 
U.S. Department of Education (USED) started a Striving Readers program. The program had 
two main goals: (a) to address the challenges of improving reading skills for middle and high 
school students who were reading below grade level; and (b) to build a scientific base to 
identify effective strategies that improve adolescent literacy skills. Striving Readers was 
geared to Title I eligible schools that had significant percentages of students reading below 
grade level and/or schools that were not meeting or at-risk of not meeting adequate yearly 
progress (AYP) requirements under the No Child Left Behind Act (NCLB). 


The program, which reflected a joint effort from the Office of Elementary and Secondary 
Education (OESE) and the Institute of Education Sciences (IES), included three key 
components: (a) supplemental literacy interventions targeted to students who were reading 
“significantly below grade level;” (b) cross-disciplinary strategies for improving adolescent 
literacy, including professional development and research-based reading and comprehension 
strategies; and (c) a required evaluation component using an experimental design (USED, 
2008). 


The first cohort of Striving Readers grantees was awarded in 2006. In 2009, USED published 
a Request for Proposal for a second cohort, with some important changes in the scope of work. 
The first cohort projects were limited to school districts and included two types of intervention: 
a school-wide model that involved professional development for all teachers in the school and 
a supplemental reading intervention for struggling readers. The second cohort competition 
requested that states put together a coalition of schools that crossed school district boundaries 
and focused solely on the supplemental reading intervention. The four-year awards would 
fund one planning year to start in school year (SY) 2009-2010, and three implementation years, 
from SY 2010-2011 to SY 2012-2013. The Illinois State Board of Education (ISBE) applied 
for and was awarded one of the eight Striving Readers grant funded as part of the second 
cohort competition. 


To prepare for the proposal, ISBE drew a list of Title I eligible high schools and invited them 
to participate in the project. Among the volunteer schools, ISBE selected six high schools in 
four school districts. The participating high schools included Danville, Eisenhower, Kankakee, 
Lanphier, MacArthur, and Springfield Southeast. Danville is the high school for the Danville 
Community Consolidated School District (DCCSD) #118. Located on the eastern boundary of 
the state, DCCSD serves about 6,000 students a year. Kankakee is the high school for 
Kankakee School District (KSD) # 111, located on the northeast part of the state near Chicago, 
and serving about 5,000 students a year. Decatur School District (DSD) # 61 was represented 
by its two high schools: Eisenhower and MacArthur. DSD, centrally located, serves about 
8,000 students a year. Springfield School District (SSD) # 186, located in the state capital, 
serves about 14,000 students a year and included two of its three high schools in the project: 
Lanphier and Springfield Southeast. 


ISBE staff also conducted a review of existing research-based supplemental adolescent reading 
programs, and decided to use Passport Reading Journeys III (PRJ III). PRJ comprises a series 
of reading interventions for students from grade 6 through 9. PRJ II/ is a supplemental reading 
intervention that targets ninth grade students who are reading below grade level. The 
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intervention incorporates whole group, small group, computer-assisted instruction, and 
individual practice, and offers a series of research-based strategies including: explicit 
vocabulary instruction, direct and explicit comprehension instruction, opportunities for 
extended discussion of text meaning and interpretation, instruction in reading foundational 
skills, and writing. 


The lessons are organized in Expeditions that focus on themes of interest for adolescent 
readers. The intervention also includes a library of fiction and nonfiction books and DVDs that 
are intended to improve student motivation and engagement in literacy learning. PRJ II 
provides a standard protocol, easy-to-follow lesson plans, an assessment system, and 
supporting materials for teachers and students. Differentiated strategies address the needs of a 
diverse student population, including students with disabilities and English language learners. 
The intervention is based on findings from reading research, is being implemented in many 
school districts nationwide, and has been studied through the use of quasi-experimental design 
(Denton, 2008; Shneyderman, 2006). However, no experimental study had been conducted, 
particularly with high school students. 


The Illinois Striving Readers (ISR) project focused on ninth grade students who scored at the 
bottom two quartiles on the state assessment (grade 8 EXPLORE®). A total of 855 students 
participated in the project. Of these, 427 students were randomly assigned to the treatment 
group, with 428 students going to the control group. 


This report presents findings from the first year of implementation of the Illinois Striving 
Readers. The report is divided into three parts. Part I describes the intervention as proposed 
by the developers and the project’s logic model. Part IT discusses findings from the first 
implementation year. Part III presents the analysis of the intervention’s impact on student 
academic performance, as measured by standardized assessments. Appendix A includes a 
summary statistics for the outcome measures, while Appendix B includes the forms used for 
data collection. 
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PART I: INTERVENTION AND LOGIC MODEL 


Description of the intervention model!’ 
Classroom Intervention 


Passport Reading Journeys (PRJ), published by Cambium Learning Group (Cambium), is a 
supplemental reading intervention that incorporates whole group, small group, computer-aided 
instruction, and individual instruction to support adolescent struggling readers. Across grade 
levels, the intervention maintains the same structure but the content and reading level change. 
The intervention is called PRJ Beginnings for sixth graders; PRJ I for seventh graders; PRJ IT 
for eighth graders, and PRJ I/I for students in ninth grade. PRJ I/I was the intervention 
implemented in the ISR project. 


The intervention encompasses daily, 50-minute lessons that provide explicit, systematic 
instruction in critical reading skills on a topic related to science or social studies. The lessons 
are organized in Expeditions for a total of 14 Expeditions that are taught within one school 
year. Each Expedition is organized in two week, ten-lesson routines to facilitate teacher-led 
instruction and students’ independent practice. Lessons one, three, six, and eight are organized 
around whole-group instruction in which students are introduced to new vocabulary and a new 
reading passage. After whole group instruction, students can individually practice vocabulary 
using the online technology component (VocabJourney) and/or select books for independent 
reading. VocabJourney is designed to enhance vocabulary and comprehension skills taught in 
PRJ III while allowing students to progress through activities in a differentiated fashion. 
Lessons two, four, seven, and nine include whole-group review of the previous day’s 
instruction and the opportunity for students to re-read the passage to build fluency, 
independently or with a partner. During this period of independent or small-group structured 
practice, the teachers are expected to work intensively with students who present specific 
instructional needs. Students spend lessons five and ten of the Expedition on independent or 
paired practice in a variety of activities intended to review, extend, or assess previous learning. 
Teachers may do any or all of the activities selected from a menu provided by the publisher. 
The choice is expected to reflect students’ learning needs. Since re-teaching may be necessary, 
lessons five and ten are intended to extend across multiple days to allow teachers to adequately 
address individual student needs. Cambium staff, in consultation with the ISR Director, set a 
two day limit for lessons five and ten to promote implementation consistency across the state. 
Therefore, for this project, the teachers were expected to complete the ten lessons that 
comprise each Expedition within twelve days. 


Core instructional elements in reading 


PRJ III aims at mixing flexibility and intensity. That is, the intervention strives to be flexible 
enough to meet the needs of older struggling readers, while intense enough to accelerate 
development of their literacy skills. Therefore, like the earlier programs, PRJ IIT blends 
reading foundational skills, vocabulary instruction, direct and explicit comprehension strategy 


' This section was reviewed by Cambium Learning Group staff for accuracy. 
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instruction, text meaning and interpretation, and writing. Alternatively, different from the 
other PRJ programs, it has less scripted language and more opportunities to engage students in 
conversation connected to the Expedition topics. The intervention is based on reading research 
and research in learning, including works from Baker, Simmons, & Kame’ enui (2004), Beck, 
McKeown, & Kucan (2002), Biancarosa, & Snow (2006), Deshler, Palincsar, Biancarosa, & 
Nair (2007), Gersten, Fuchs, Williams, & Baker (2001), Graham & Perin (2007), Marzano 
(2004), Mastropieri, Scruggs, & Graetz (2003), Scammacca, Roberts, Vaughn, Edmonds, 
Wexler, Reutebuch, and Torgesen (2007), and Schatschneider, Buck, Torgesen, Wagner, 
Hassler, Hecht, & Powell-Smith (2004). 


Explicit vocabulary instruction focuses on words the students must understand in order to 
comprehend each text segment and make connections between the words. Vocabulary 
instruction also involves context clues, word parts, including compound words, morphology, 
dictionary skills, high-frequency words, content-related words, synonyms and antonyms, 
multiple-meaning words, and homophones. A planned sequence of vocabulary skills and 
multiple exposures of high-utility words are meshed within the passages, comprehension 
activities, and text discussions. Affixes and roots are explicitly taught to students in a 
sequential pattern that is supported by the identified words in the passages. VocabJourney 
provides self-paced practice on vocabulary and comprehension skills, while teaching additional 
academic vocabulary and providing vocabulary support for English language learners. 
Students work individually and the software monitors student responses and adapts instruction 
to meet individual student needs. 


Instruction in reading foundational skills is provided through the advanced word study portion 
of the daily Expedition lessons and the supplemental word study component of the intervention 
program. The supplemental word study component is geared to offer more individualized 
support to students who have difficulty with decoding and fluency. The thirty word study 
lessons comprise phonemic awareness, recognition of high-frequency words and irregular 
words, as well as sight words, spelling, and phonics. These word study lessons may be 
provided prior to implementing the first Expedition lesson or on alternate days once the 
intervention sequence has begun. 


Direct and explicit comprehension strategies are woven into instruction to help students 
develop skills that are traditionally lacking among striving readers, such as making and 
confirming predictions, reading charts and graphs, identifying or stating main ideas, generating 
questions, summarizing, and making inferences (Baumann, Font, Edwards, & Boland, 2005). 
Comprehension skills are taught explicitly and applied to expository passages in both the text 
and VocabJourney. Live Ink, available in VocabJourney, allows students to read online 
passages and anthology passages in shorter chunks. Students also examine organizational text 
features that serve as frames for information and logical links between ideas. Comprehension 
strategies are scaffolded in three stages: teacher modeling, teacher assistance with student 
practice, and student independence. The stages represent a gradual shift in responsibility for 
learning from the teacher to the students. Direct instruction includes modeling in which the 
teacher reads aloud to show students how to use the reading strategies. A thinking aloud 
method is employed to make thought processes more transparent to students. Modeling is 
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followed with direct, guided practice and self-assessment to enable students to apply the newly 
learned skills and strategies in a variety of texts that cover varying levels of reading ability. 


Opportunities for extended discussion of text meaning and interpretation are elicited through 
questions posed by the teacher during and after reading. In the first reading of the selection, 
the teacher asks literal comprehension questions to ensure understanding and to model the 
metacognitive process of self-monitoring. After students complete their reading, the teacher 
asks critical thinking questions that reflect the various levels of the revised Bloom’s Taxonomy 
(Anderson & Krathwohl, 2001). This teacher-directed questioning is integrated with student- 
generated questions as a key reading comprehension strategy during reading and a way for 
students to monitor and deepen their understanding of the text. 


PRJ III includes a two-fold approach to writing. First, it uses writing in response to reading, 
which helps students check their understanding, reinforces returning to the text for more 
information, and sharpens critical thinking skills. Every Expedition integrates writing practice 
and instruction. Secondly, it offers a writing activity option during lesson five and/or ten of 
each Expedition that extends the comprehension skills and content into a writing topic. These 
lessons are designed to help students develop writing proficiency. The writing activity 
includes practice in pre-writing strategies, elaboration, word choice, sentence fluency, 
organization, and writing paragraphs. 


Motivation and engagement in literacy 


To improve student motivation and engagement in literacy learning, PRJ III offers a library as 
part of its instructional materials. The library contains novels and texts that allow for partnered 
and guided reading. The books are Lexile-leveled, so that students can choose a book at his or 
her skill level. The fiction and non-fiction texts have been field-tested for high interest with 
high school students and reach across the curriculum to foster literacy development in social 
studies and science content areas. Career excerpts are highlighted throughout the anthology. The 
characters, content, and activities target students who represent diverse cultural and linguistic 
groups. Examples of topics for the Expeditions include Music without Instruments, Military 
Medical Innovations, and The Multicultural Search for Beauty. DVD segments are presented 
before and after each Expedition to provide background knowledge and create the foundation 
for understanding of content. Each video segment is hosted by a teen who asks probing 
questions, highlights essential content-area vocabulary words, makes relevant connections to 
students’ lives, and engages them in thinking about the topics at hand. 


Use of technology 


Technology is incorporated into PRJ III through the VocabJourney. In addition, lessons one, 
six, and nine include video technology in the form of DVD segments. VocabJourney is a web- 
based interactive program that provides teachers with more flexibility to individualize 
instruction. It is comprised of three main components — Acquire, Achieve, and Connect. In 
Acquire, students learn new words and practice previously learned works. In Achieve, 
students deepen word knowledge through word games, flash cards, and exploring multiple 
word meanings. In Connect, students study words in categories by content or topic. Research 
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has found that computer-assisted reading instruction helps struggling readers by providing 
individualized instruction, immediate feedback, a motivating learning environment, a method 
for monitoring student progress, and a way to maintain student interest (Kim, 2002; Kim, 
Vaughn, Klingner, Woodrugg, Reutebuch & Kouzekanani, 2006). 


Professional Development 


Cambium Learning Group offers diverse professional development activities for the teachers 
that include launch training, online product training, ongoing consultative support, coursework 
on adolescent literacy, and data analysis meetings. The launch training, the online product 
training, and the online support are part of the PRJ III package, while the other activities 
depend on separate contracts between the schools or school districts and the developer. 


Group professional development 


The launch training is traditionally conducted as two eight-hour sessions intended to prepare 
teachers to implement the intervention with fidelity. Participants learn about the intervention, 
and are instructed in specific practices, such as administering the assessment measures, 
grouping students, setting up their classrooms, structuring small and large group instruction, 
and using intervention materials. The training includes time for practice of lesson delivery and 
instruction in VocabJourney, the technology component of the intervention. Training on the 
Voyager data management system (VPORT) and classroom management are also included. 


During launch training, participants are invited to observe and reflect as the trainer 
demonstrates a lesson. Following the demonstration, participants have opportunities to 
practice teaching the lesson. They regroup at the end of the training to debrief and plan next 
steps. Materials include a DVD showing footage of classroom instruction, illustrations of 
program features, and the measures to practice administering and scoring the assessments. 
Tutorial booklets introduce the key features and components of the program, present sample 
lessons at each grade level, review the assessment component, and provide suggestions for 
managing time and working with students with special needs. 


The online training modules provide instruction in a self-paced, interactive environment that 
allows teachers to search, annotate, and bookmark information. Each module includes 
curriculum, assessment, and implementation overviews, and provides links to a library of video 
segments. The modules also offer suggestions on classroom management and on 
understanding Lexile levels. At the conclusion of each section, the teachers take a quiz to 
check knowledge gained. They can redo the modules to improve knowledge, or come back to 
them later to refresh information. 


Coursework on adolescent literacy is delivered through VoyagerU, Cambium’s professional 
development arm. Two 15-hour courses present foundational information about adolescent 
literacy, define the specific reading skills the students need in order to master each academic 
subject, and identify the best strategies to help middle school students develop their reading 
comprehension skills in these subjects. These courses were developed by Deborah Reed, 
principal investigator and project manager for the Texas Adolescent Literacy Academies, 
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Diane Lapp, Distinguished Professor of Education, San Diego State University and a member 
of the International Reading Association Hall of Fame, and Douglas Fisher, professor of 
language and literacy education at San Diego State University and co-director for the Center 
for the Advancement of Reading at the California State University. VoyagerU courses are not 
part of the regular PRJ III package. Details about the professional development planned for 
ISR are described in the Logic Model section. 


Individual supports 


Cambium offers individualized supports for teachers who are implementing PRJ /// through 
trained experts, the Voyager Implementation Specialist (VIS). The VIS conducts visits to each 
participating schools to observe how the intervention is being implemented. The frequency of 
the visits is dictated by teachers’ needs and the contract established between Cambium and the 
school or school district. The VIS reviews student data with the teacher on an ongoing basis to 
accurately formulate prescriptive technical assistance, which must be geared towards each 
teacher’s needs. At the end of each site visit, the VIS conducts a debriefing section with each 
observed teacher, the building principal, and other designated parties. As the teachers began to 
implement the intervention, they are encouraged to contact the assigned VIS whenever needed, 
either by telephone or e-mail. 


Assessments 


The assessment system within PRJ I/I includes benchmark assessments, semester exams, end- 
of-lesson assessments, progress monitoring, and student self-assessments through 
VocabJourney. The Reading Benchmark assessments are based upon Lexiles that allow 
educators to quickly estimate expected reading comprehension and monitor progress. Lexile, 
developed by MetaMetrics, Inc., is a measure of the difficulty of comprehension of a text 
(Stenner, 2001; Stenner & Wright, 2004). The measure, based on calculations of word 
frequency and sentence length, is presented on a scale that ranges from OL to 2,000L. Text 
measures at or below OL (zero Lexiles), are reported as BR (Beginning Reader). 


The Reading Benchmark assessments were developed using the Rasch one-parameter item 
response theory model to relate a reader's ability with the difficulty of the items. The primary 
sources of validity evidence for Lexiles comes from an examination of the content of the PRJ 
assessments, and the degree to which the assessments measure reading comprehension 
(Lennon & Burdick, 2004). The Reading Benchmarks are expected to be administered in a 
whole group format three times per school year during specified benchmark assessment 
periods to assess comprehension (MetaMetrics, 2009). The Reading Benchmark I is 
administered at the beginning of the school year to place students in the appropriate level of 
reading materials. The Reading Benchmarks II and IIT, conducted during the school year, are 
used to monitor student progress on vocabulary and comprehension. 


Formative assessments are also administered at the end of each Expedition. These criterion- 


referenced tests measure comprehension and vocabulary skills that have been taught during the 
Expedition lesson series. Additionally, student self-assessments are available through the 
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VocabJourney reports, which provide students with guided feedback on their responses. The 
teachers can review the feedback provided to students by logging into VPORT. 


Based on student performance on these assessments, the teachers are directed to re-teaching 
opportunities that are targeted to specific skills where students have demonstrated difficulty. 
Additionally, criterion-referenced semester exams assess students’ ability to apply the 
vocabulary and comprehension strategies taught through the Expeditions. 


Target population 


PRJ Ill is geared to students in grade 9 who are defined as struggling readers by their schools. 
The intervention incorporates a number of differentiated strategies that are designed to address 
students with a broad range of reading levels, including students who have limited English 
proficiency, and students with disabilities who can be served through group instruction. 


Desired characteristics of the teachers 


PRJ III reflects a highly structured intervention. Each teacher receives a teacher’s guide that 
includes an explanation of the intervention, the goals, and the scope and sequence of each 
component followed by detailed guidelines on how the lesson must be taught. The teacher is 
expected to follow the guidelines and maintain the scope and sequence of each lessons’ 
components. Small variations within the lessons are allowed to address differences in class 
period and students’ needs, as explained in JSR Logic Model. 


Decisions about hiring teachers are left to the local education agencies (LEAs). The 
intervention’s scripted format and the professional development offered are intended to 
facilitate instruction by experienced and non-experienced teachers alike. For teachers who do 
not have a reading background, Cambium provides additional training on reading through its 
professional development branch (VoyagerU). 


Desired characteristics of the classrooms 


Cambium’s requirements for PRJ I/I classrooms include appropriate space for small group 
instruction and storage and use of material connected to the lessons, including teacher guide 
books, students’ workbooks, and the library. Additionally, the classrooms are required to have 
a DVD projector and computers. Cambium recommends a maximum of 20 students per 
classrooms. 


Recommended intensity for the students 


PR/J IIT is to be taught daily in a 50-minute period class within one school year. As more 
schools are adopting the 90 minute block time, the intervention has been adapted to allow for 
teachers to cover two shortened lessons within the block period. The pacing of the lesson 
should strike a balance between the expected one lesson per period and students’ needs. If the 
lesson cannot be completed within the allotted period, the teachers are instructed to continue it 
the following day, starting from the point they stopped the day before. Reducing writing time 
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is an allowable strategy to accommodate the pacing, but reducing reading time is not 
recommended. The teachers are not expected to complete all 14 Expeditions within the year, 
although they should try to cover as many as possible. 


Illinois Striving Readers (ISR) Logic Model 


Overview 


Figure | displays a graphic representation of the logic model proposed for the study of the ISR 
project. The model comprised two components and two outcomes. The components included 
the professional development model and the classroom instruction model. These two 
components followed as close as possible to the model proposed by Cambium, with variations 
that addressed the specific needs of participating schools and ISBE’s requirements for the 
project. 


As seen in the graphic, the professional development model contained a total of 14 hours, 
which included eight hours of launch training and six hours for data training. For the launch 
training, each RIT met one-on-one with the VIS for one eight-hour intensive training session. 
The data training, which occurred in March 2011, brought all the RITs together for six hours of 
data review and discussions on using data to inform instruction. The RITs could also pursue 
16 hours of online modules to reinforce or expand the information obtained during the launch 
training. These were optional hours that depended on the RIT’s perceived need. Additionally, 
ISBE required teachers to attend the three-day annual convention of the International Reading 
Association (IRA). The second component, the classroom instruction model, focused on the 
delivery of the intervention and assessments with fidelity. 


The personnel resources available to support the project implementation included 11 educators 
located within the schools, the local education agencies, and the state education agency. Each 
participant had defined roles and responsibilities within the project and in relation to the study 
conducted by RMC, as explained below. 


At the state level, the grant funded a 0.5 Full Time Equivalent (FTE) Project Director position 
to oversee the project. The ISBE Project Director, who worked under direct supervision from 
the Division Administrator of Curriculum and Instruction, had a number of coordination and 
monitoring responsibilities, including: communicating and collaborating with the participating 
school districts, Cambium staff, the external evaluators, and the USED representatives; 
approving and monitoring the use of grant funds by the participant districts and schools; and 
providing support to the RITs to ensure that the program was implemented with fidelity. 


The grant also funded a LEA Project Coordinator at 0.10 FTE for each school district. At the 
district, the LEA Coordinators had four main responsibilities: (1) overseeing the hiring 
procedures for the RITs, (2) coordinating project-related activities with the ISBE Project 
Director, Cambium, and the evaluators; (3) supervising the day-to-day implementation of the 
intervention within the district; and (4) providing student-level outcome data to the evaluators 
for the random assignment and the impact study. 
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Figure 1. Logic Model for the study of the Illinois Striving Readers 


Strategies (Model): Voyager Passport Journeys LIT 
Professional Development Model Classroom Model (9" grade classrooms] 


Year 2: Structure 
“*Summer 2010: Launch training (8 “*Year-long, daily 50 - minute lessons or every other day, 90-min block 
hours; required) **14 Expeditions; each Expedition divided into ten lessons, as such: 
**School year: Lessons 1 and 3 
o Online module (16 hrs. optional) o Introduction to Expedition - day 1 only (discuss probing questions) 
o In-school coaching (40 hrs. Before reading (introduce and practice vocabulary) 
maximum) During reading (reading related to the topic) 
o Assessment training (6 hrs. After reading (check comprehension) 
required) Independent study= practice vocabulary online; reading 
o Attendance to the International independently 
Reading Association (IRA) Annual o ELL= extend and practice 
Convention (24 hrs. required) Lessons 2 and 4 
Years 3-4: o Prepare to reread = review and practice (vocabulary, comprehension)| 
** New teachers = as above practice using context clues; build new words using roots 
“* Returning teachers o Reread = review and practice finding implicit main idea and details; 
o IRAconvention (24 hrs.) write a paragraph 
o In-school coaching (25 hours) o ELL = extend and practice 
“* Attendees: Lessons 5 and 10 
o Teachers= mandatory o Review, extend and assess 
participation Content 


Outcomes 


Short-term 


At least 50% of 
the treatment 
students will 
improve at least 

one reading level, 

as measured by 
standardized 


[e) 
2) 
[e) 
a assessments. 


Long-term 


Gains in 9" grade 
persist as students 
move to upper 


o Project leadership = not required ** Lexile-leveled, focused on adolescent themes related to science, social grades ah 
Provider: studies, and careers measured by 
standardized 


** 2 Voyager Implementation Specialists | Assessment 
(Under guidance of the Vice- “*Reading Benchmarks I - placement (September) 
President for Implementation “*Benchmarks IJ and III - progress on fluency(January and May) 
Services for the Northeastern “*Comprehension and vocabulary assessment - Expedition days 5 and 10 
Region) “*Semester Exams - middle and end of the year (end of Expedition 7 and 14) 


assessments 


Personnel 


6 FTE Reading Intervention Teachers 4— Principal or Assistant Principal <——— 4 LEA Project Coordinators <—— ISR Project Director 
Classroom School LEA ISBE 


Technology and supplies 
“* Grant-funded: computers, DVD projectors 
“* Cambium : books and magazines in print, audio and e-books, teacher guides; student workbooks; other supporting materials 


RESOURCESS 


Illinois Striving Readers, Implementation Year One Report 11 


At each of the six participant schools, a building administrator worked with the LEA Project 
Coordinator and the evaluators to facilitate and monitor implementation and ensure the fidelity 
of the random assignment process. Each school hired one full time equivalent RIT who was 
dedicated to the intervention for at least 80 percent of the time. During the remaining 20 percent 
of time, the RIT could be assigned to other duties, such as coaching teachers for grades 10-12 on 
strategies for reading across the curriculum. 


The PRJ developer, Cambium Learning Group, was responsible for the professional 
development component of the project. Cambium-designated coaches, or Voyager Intervention 
Specialists (VISs), worked under the supervision of the Vice-President for Implementation 
Services for the Northeastern Region. The VISs were responsible for providing professional 
development and coaching to the RITs; supervising the work of the RITs at each school through 
the use of Cambium monitoring instruments; and providing feedback to the RITs and building 
administrators to ensure that the intervention was being implemented with fidelity. 


Materials to support the intervention included PRJ //I instructional materials and a library of 
DVDs and books, provided by Cambium. Grant funds were used to buy computers and DVD 
projectors, while the schools’ regular library expanded the students’ choices of reading material. 


The project included a short-term, yearly outcome, and a long-term outcome that would be 
measured through a growth model design. For the short-term outcome, ISBE expected that 50 
percent or more of the students who participated in the intervention would improve at least one 
grade level as measured by standardized assessments at the end of each intervention year. The 
long-term outcome was that treatment students would show retention of gains in reading skills 
when taking the statewide assessment in grade 11. 


RMC Research Corporation, the external evaluator for the project, was responsible for 
developing the instruments for data collection and collecting data for the implementation 
evaluation through interviews, review of documents, and classroom observations. RMC was 
also responsible for the random assignment of students and conducting the impact analysis. The 
interaction between implementation and the ISR study is graphically represented in Figure 2, on 
the next page. 
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Figure 2. Illinois Striving Readers —Implementation and study interaction 
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ISR professional development model 


Table 1 summarizes the initial professional development activities planned for the RITs and key 
project participants. Between August 2010 and the spring of 2011, the RITs were to receive 14 
hours of required PRJ-related professional development. Additionally, the RITs were expected 
to attend the IRA Annual Convention, which occurred on May 8-11, 2011 at the Orange County 
Convention Center, in Orlando, Florida. Grant funds paid for airplane tickets, lodging, per diem, 
and conference fees for all six RITs. RITs also received a one-year IRA membership that would 
be renewed throughout the grant period. 


Table 1. Professional development activities related to the Illinois Striving Readers (summer 
2010 - spring 2011) 


Attendees | Hours 
Individual | Cambium Learning Introduction to the program; RITs 
Group (VIS) assessments, technology, 
classroom management. 
Individual | Cambium Learning One-day meeting for training RITs 
Group (VIS) on administering, analyzing 
and using assessments for 
instruction. 


Whole International Reading | Grant support for attendance RITs 24 
group Association (IRA) to the annual IRA convention ISBE Project 


Director 
Required hours 
Self- Cambium Learning PRJ modules supplement 
paced, Group launch training Topics: 
online, curriculum, assessment, 
optional implementation, classroom 
management, understanding 
Lexile levels, and other topics. 
Individual | Voyager On-site support includes 
support Implementation classroom observations and 
(coach) Specialists feedback, lesson modeling and 
use of data for instruction. 


ISR Director Community of Practice 


iw) me | Go 
N Nn O| oo 


— 


RITs to be hired in implementation years 2 and 3 would receive similar training, but teachers that 
remained in the project would be offered the online modules, if needed, and a maximum of 25 
hours of individualized coaching provided by the VIS. The ISBE Project Director, the LEA 
Project Coordinators, and the school administrators were not required to participate in any of the 
ISR-sponsored professional development activities. 


Two VIS were contracted to provide a maximum of five days a year of on-site coaching for the 
RITs for a total of 25 hours. The VIS worked under the supervision of Cambium’s Vice 
President for Implementation Services in the Northeast Region. Coaching activities included 
modeling lessons, observing lessons and providing feedback, discussing student data and using 
data to plan the lessons, asking clarifying questions, and supporting the RITs as they 
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implemented the intervention with their students. The VISs were also available to address 
questions and concerns as needed via conference call and e-mail. 


The topic of the individual on-site coaching support was tailored to the teachers’ needs, but the 
coaching model followed a similar format in all participant schools. First, the VIS observed a 
lesson taught by the interventionist. The VIS would then model a lesson, followed by a 
debriefing session during which the VIS discussed the observation with the interventionist and 
made recommendations for improvement. The VIS observed the interventionist teaching another 
lesson on a different day to see if he or she was incorporating the recommendations. During the 
coaching visits, the VIS used Cambium’s Index of Fidelity of Implementation (IFI) to assess 
how close the intervention implemented in the classroom was to the model intervention. The IFI 
was used as the foundation for the classroom observation instrument developed by the 
evaluators. 


The ISBE Project Director also maintained monthly conference calls with the RITs using a 
Communities of Practice model, whereby the RITs exchanged information about their work and 
discussed a journal article assigned by the Project Director, with a focus on the classroom 
application of the article. Additionally, as part of their supervisory responsibilities, principals 
conducted regular in-class visits in conjunction with the VIS visits. Also, the LEA Project 
Coordinators visited the classrooms as needed to observe the implementation of the project. In- 
class visits and support from school principals, LEA coordinators, and the ISBE Project Director 
had the purpose of monitoring and supporting the RITs. 


Planned classroom instruction 
Adaptations to the classroom model 


The planned classroom model followed the developer’s model, as described above (see 
Description of the intervention model). Teachers in schools with 90-minute block scheduling 
were recommended to present two lessons in one day, thus modifying the 50 minute lessons to 
45 minutes each. The total number of instructional hours remained the same. Adaptations were 
discussed during the planning year to address the different class periods at the participating 
schools. For instance, Cambium suggested the RITs eliminate some of the writing exercises 
within each task in the Expedition to maintain the pacing of the lessons, placing the focus on the 
reading process rather than the writing material. Another suggestion was to use at least one of 
the extra days in lessons 5 and 10 to initiate a new lesson or start a new Expedition if students 
were showing strong results in the lesson-specific assessments. 


Experiences for control group students 
PRJ III instruction was provided in periods that coincided with time reserved for electives to 
ensure that students participating in the study could attend the core content area classes offered 


to all students in the same grade level. Students not attending the intervention had the option to 
enroll in electives that were not related to reading instruction. 
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Remediation classes 


Illinois grade 9 students do not participate in statewide assessments used for purposes of 
determine AYP. Content area teachers, including English teachers, may individualize instruction 
during their regular classroom time through differentiated instruction strategies, such as small 
group instruction. These strategies, when available, are provided to all students in the classroom 
according to their needs. Therefore, no additional instruction was being provided to one group in 
the study (e.g. control group) at the exclusion of the other group (e.g. treatment students). More 
importantly, no reading instruction outside PRJ II] was being provided at the ISR schools. 


Teachers characteristics 


By the summer of 2010, a total of six Full Time Equivalent (FTE) Reading Intervention Teachers 
(RITs) had been hired to provide instruction in PRJ II, one per school. The hiring of the RITs 
followed the process used by the school districts to hire their regular teaching staff. The position 
was announced in local newspapers, and the applicants were interviewed by a panel that included 
the school principals, who made the final decision. The RITs were required to have a valid 
Illinois teaching license with a reading endorsement, two to three years of teaching experience, 
an understanding of the Response to Intervention process, and proven classroom management 
skills. As part of the contract, the RITs were required to dedicate at least 80 percent of their time 
to the intervention, attend all of the professional development activities related to the 
intervention, and implement PRJ I// with fidelity. 


Classroom characteristics 


The teachers were assigned permanent classrooms that were expected to have enough space to 
conduct whole group and small group instruction. Computers and DVD projectors were 
available for instruction. ISBE staff held a discussion regarding the maximum number of 
students per classroom during the planning year, and made the decision to allow up to 25 
students per classrooms. 


Assessments 


Table 2, on the next page, lists the assessments used for the ISR project. As described in Part I, 
Description of the Intervention Model, PRJ II uses a variety of assessments to identify students’ 
needs and inform instruction. During the launch training, the teachers were instructed in how to 
administer the assessments, analyze data, and use the information to individualize instruction. 
The assessments used in the ISR project included: 


e Reading Benchmark I, administered at the beginning of the year to place students with 
the appropriate level of reading materials; 

e Reading Benchmarks II and III, administered in January and May and used for 
monitoring and informing instruction; 

¢ Comprehension and vocabulary assessments, administered during lessons 5 and 10 of 
each Expedition to measure comprehension and vocabulary skills; 
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e Semester exams, criterion-referenced assessments that focus on mastery of skills and 
content taught, and are administered online at the end of Expeditions 7 and 14; and 

e Online vocabulary technology self-assessments, used by students to track progress on 
vocabulary, comprehension and content-specific text. 


Table 2. Illinois Striving Readers — List of assessments 


Instruction 


Placement 
and 
Monitoring 


Reading Benchmark I 
Placement 


September 


Place students at 
appropriate reading 
levels 


Reading Benchmarks 
II and II 


January and May 


Monitor student 
progress on 

vocabulary and 
comprehension 


Semester Exams 


End of Expeditions 7 
and 14 


Impact study 


Monitor student 
progress 


Eligibility 


Grade 8 EXPLORE® 


Spring previous school 
year 


Determine eligibility 
to the study 


Grade 9 EXPLORE® 


May of the school year 


being studied 
Gates-MacGinitie Reading Early fall and late spring 
Tests (GMRT) 4th Edition — of each implementation 
grade 9 year 


Outcomes Impact study 


Two assessments were used to measure student performance: The Gates MacGinitie (GMRT®) 
4" edition for grade 9, and the Grade 9 EXPLORE.” GMRT,® from Riverside Publishing, is a 
nationally-normed reading assessment that has well-established psychometric qualities, as 
documented in the 2002 technical manual (MacGinitie, MacGinitie & Dryer, 2002). 
Approximately 3,600 students in grade 9 participated in the standardization. Kuder-Richardson 
Formula 20 (K-R-20) internal reliability coefficients were 0.92 and 0.93 for the raw scores for 
the ninth grade students who took the Level 7/9 test (Form S) in the fall and spring, respectively. 
The test was re-normed during SY 2005-2006 (Maria & Hughes, 2008). For the ISR project, 
grade 9 GMRT® was to be applied twice a year, early in the fall and in late spring of each 
implementation year. 


Grade 9 EXPLORE® is a component of the ACT testing system, published by ACT, Inc. This 
criterion-referenced test measures academic achievement in English, mathematics, reading and 
science using a multiple choice format. Details about the test development and psychometrics 
are found in the technical manual (ACT, 2011). 


ISR eligible students 
Students eligible for the project were defined as those who scored at the two lowest quartiles in 


the grade 8 EXPLORE® reading assessment. These students were distributed across six high 
schools located in four school districts in Illinois. Table 3 displays the number of students 
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enrolled in the participant schools at the end of SY 2008-2009, the percentage of minority and 
low income students, the mobility rate, and the number of students who scored in the two lowest 
quartiles and were to be enrolled in the participant schools the following school year. For the rest 
of this report, the schools will be named by randomly attributed numbers to maintain their 
privacy. 


Table 3. Illinois Striving Readers schools (SY 2008-2009) 


High Enrollment | Minority Beas Mobility besa 


Schools (N) (%) (%) rate (N)* 


School District 


Danville Community 


Consolidated Danville 


Eisenhower 
MacArthur 
Kankakee Kankakee 


Decatur 


Lanphier 
Springfield Springfield 
Southeast 


*Number of 8" graders who scored in the two lowest quartiles in the SY 2008-09 statewide assessment and were expected in the 
Striving Readers schools 
Source: Illinois State Board of Education 


As displayed in the table, in SY 2008-2009, before the beginning of the intervention, these six 
schools served a total of 7,568 students in grades 9 through 12. On average, students defined as 
low income comprised about 58 percent of the schools’ population. Mobility rates varied from 
30 to 38 percent. Students scoring 12 or below on the reading component of the grade 8 
EXPLORE® were eligible to participate in the project. The last column on the table displays the 
numbers of potential struggling readers that expected in the participant schools during SY 2009- 
2010. Two exclusion criteria were adopted: students with an Individual Education Plan (IEP) 
that precluded their participation in group instruction, and students whose parents requested their 
children be exempt from the study. Part III: Impact Study details the process used to determine 
eligibility and conduct the random assignment. 


Expected student outcomes 


ISBE proposed two goals and seven objectives in its application to USED. The goals and 
objectives included: 


Goal 1: Foster reading improvement in grade 9 striving readers 
e Objective 1.1: By the end of Project Year I, project staff and the external evaluator, in 
consultation with the U.S. Department of Education, will have developed implementation 
and evaluation plans for the Illinois Striving Readers project. 
e Objective 1.2: By the end of Project Year 4, provide Passport Journeys III supplemental 
reading intervention classes to 1,350 grade 9 striving readers in six participating high 
schools. 
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e Objective 1.3: By the end of each school year during Project Years 2-4, 50 percent of 
students in the intervention cohorts will demonstrate a gain in reading achievement, at a 
minimum, of one grade level or its equivalent. 

¢ Objective 1.4: By the end of Project Year 4, 75 percent of students in the Passport 
Reading Journeys III cohorts will score at or above proficient on the state reading 
assessment. 


Goal 2: Contribute to the scientific research base for identifying and replicating supplemental 
reading programs that are effective in improving adolescent literacy. 

e Objective 2.1: By the end of Project Year 4, the external evaluator will conduct a project 
evaluation that assesses the effectiveness of the implementation of Journeys III in the six 
participating high schools. 

e Objective 2.2: By the end of Project Year 4, the external evaluator will conduct a project 
evaluation that assesses the effectiveness of Passport Journeys III in raising the literacy 
levels of grade 9 striving readers in the six participating high schools. 

e Objective 2.3: By the end of Project Year 4, disseminate research findings on the 
effectiveness of the intervention in raising the literacy levels of grade 9 striving readers. 


Planning Year (2009-2010) 


After the USED award was confirmed, ISBE issued a Request for Proposal (RFP) for the 
evaluation of their Striving Readers project. RMC Research Corporation (RMC) applied for and 
was awarded the contract in March 2010. The evaluators used the remainder of the planning 
year to finalize the evaluation design and visit the participating schools. The evaluators’ visits 
had three main purposes: (a) to become familiar with the schools’ procedures; (b) to understand 
how the project was to be implemented within each school; and (c) to explain the design 
(particularly the random assignment process) to school administrators and LEA representatives. 
Discussions related to how to explain the study to parents were also conducted and it was 
decided that the schools would follow their existing procedures for parental consent, with 
support from the evaluators (if needed). Each school nominated a staff member — either an 
assistant principal or the school scheduler — to be the liaison to evaluators for the random 
assignment, while the LEA Project Coordinators assumed the responsibility for providing the 
evaluators with assessment data to be used in the impact study. 
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PART II: IMPLEMENTATION STUDY 


Study Design 
Overview 


The evaluation of the ISR implementation aimed to assess how close the implemented 
intervention was to the PRJ III model. The implementation study was to be used as a descriptive 
tool to further the understanding of the findings from the impact study. The intervention’s 
structure provided the framework upon which the research questions and the development of 
instruments for data collection were built. As discussed in Part I, PRJ IIT is a highly-structured 
intervention in which the RITs are expected to attend all of the required professional 
development activities and to follow a scripted guide that details what, how, and when they will 
teach. Diversions from the model are not expected or welcomed, except for minor adaptations to 
adjust the expected pacing within allocated classroom time. 


The evaluation strived to be as unobtrusive as possible to reduce the possibility of a “Hawthorne 
effect” (Gillespie, 1991). The evaluators had no direct contact with the students involved in the 
project, except for silent classroom observations to which students are accustomed as a part of 
regular administrative visits. 


Research questions 


The implementation evaluation focused on fidelity to the intervention model proposed by the 
developers and incorporated in the ISR Logic Model. The implementation study was guided by 
one broad question: 


e To what extent did the implementation of PRJ //T in the Illinois Striving Reader project 
schools reproduce the developer’s model? 


This broad question was then divided into three sub-sets of questions: 
e Professional development: 
o What types and how many hours of professional development were offered to the 
RITs? 
o What types and how many hours of professional development did the RITs 
attend? 
o How many hours of coaching support did the RIT receive? 
e Classroom instruction: 
o How many hours of classroom instruction were planned? 
o How many hours were provided? 
o To what extent did the RITs follow the model guidelines? 
e Influencing factors: 
o What factors facilitated the implementation? 
o What factors created barriers to the implementation? 
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Data Collection Plan 


To collect information on the professional development and classroom models, data were 
collected from the following sources: 


e Voyager Implementation Specialists — Cambium maintains logs with the types and hours 
of professional development offered, who attended, and what was covered during the 
activities. These data are accessed through VPORT. The evaluators also interviewed the 
VISs to obtain information regarding the coaching services provided and their 
perceptions about the PRJ I// implementation in the different schools, including the 
barriers and the facilitators encountered. 

e Reading Implementation Teachers — The evaluators scheduled monthly “check-ins” with 
the RITs to obtain their perspectives on project implementation. Due to difficulties in 
contacting all of the RITs by telephone on a monthly basis, beginning in early 2011, they 
could choose between telephone and online check-ins. Between September 2010 and 
May 2011, a total of eight check-ins were completed for each interventionist. 

e ISBE Project Director — An interview was conducted with the ISBE Project Director 
regarding the supports provided to the RITs and perceived barriers and facilitators to the 
implementation. 

e Classroom visits — Two evaluation teams were assigned to observe PRJ III classrooms, 
with each school being visited by one team. Each team was comprised of an evaluator 
with a reading background, who had been trained on the intervention, and an evaluator 
with a methods background. The evaluation plan included four one-day visits to each 
school, at the beginning, middle, and end of the school year. For each of the visits, the 
evaluators would conduct alternating half-day observations at each school (e.g. on day 
one, school A will be observed during the morning and school B will be observed during 
the afternoon; the following day, school A will be observed during the afternoon and 
school B will be observed during the morning, and so on). This process allowed for 
observing instruction at different times of the day and on different days of the week to 
cover the ten lessons routine that characterizes each Expedition. 


The evaluators took extensive notes during the interviews and used categorizing and connecting 
process for the analysis (Maxwell & Miller, 2008). Data were coded thematically to reflect the 
PRJ III components highlighted by the developer in its Index of Fidelity of Implementation (IFI). 
These components included: amount of instruction, quality of instruction, classroom 
management, use of assessment, and differentiation. Information from the interviews was used 
to foster understanding of the process of implementation in the different schools from the 
different actors’ perspectives, as well as to provide information regarding the barriers and 
facilitators to implementation. The check-in questionnaires, classroom observation rubrics, and 
interview protocols are included in Appendix B. 


Although the evaluators wanted to make unannounced visits, the complex reality of school 
scheduling made that impossible. To organize the visits, the evaluators obtained schedules from 
each school highlighting the weeks when the schools would be occupied with statewide tests, 
spring break, and/or other events that disrupt regular classroom instruction. Coordination with 
the RITs was also needed to avoid visits on days when class time was used for assessments. The 


Illinois Striving Readers, Implementation Year One Report 21 


evaluators visited each of the participant schools twice during SY 2010-2011, one time in 
October 2010 and the other in February 2011 for a total of 27 observations. Two other visits 
were planned for Mid-March and May 2011. However, with the cancellation of the Striving 
Readers grant, and the need to maintain funding for the data analysis, the two final visits were 
cancelled. Table 4 lists the data sources and data collection processes used in the 
implementation evaluation. 


Table 4. Implementation evaluation — data sources and data collection processes 
Source Process Purpose 
Gisseended Obtain information 


Reading teacher check-in aaclaiag taeuaion 
(RIT) : 


Open-ended 


interview january NY 


Voyager 
implementation 
specialist 

LEA Project Open-ended 
Coordinator interview 
ISBE Project Open-ended 
Director interview 
Classroom Oct, Feb — completed 


Observation Sealed rubric March, May - cancelled 


VPORT Excel spreadsheet | January/March/June 


Open-ended 
interview 


January/May 


Triangulate 
perspectives 
May (cancelled) regarding the 
implementation 


May 


Enrollment; 
benchmark scores 


Defining fidelity of implementation 
Professional development model 


Attendance to launch training and data meeting were required from all RITs. To reduce grant 
costs and RITs’ travel time, the VIS provided the training to each RIT independently and on-site. 
This process allowed for a reduction of the traditional 16 hours of launch training to no more 
than eight training hours. Statements on the quality of the training were deemed unnecessary due 
to Cambium’s quality control measures. The face-to-face PD activities were conducted by the 
developers’ trained experts and the online modules were created by reading experts and 
maintained by Cambium’s research division. 


The other required intervention-related training was the data meeting, which occurred in March 
2011. This statewide meeting focused on how to analyze assessment data and use it to 
individualize instruction. Additionally, RITs were required per contract to attend the annual 
convention of the International Reading Association (IRA), an activity that was not directly 
related to PRJ III implementation, but was part of the ISR model. Required PD hours added to 
38 hours, with 14 hours directly related to the intervention and 24 hours focused on reading 
research and practice. 
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The RITs also had available a maximum of 25 hours of coaching and 16 hours of online 
modules. Usage of these hours depended on the willingness and need of the interventionist; 
therefore, these hours were included as extra hours for the fidelity index. Additionally, the ISBE 
Project Director offered a forum for discussion and exchange of information that were included 
in the extra hours count. 


The score for the fidelity of implementation of the professional development model was 
calculated as the number of hours attended relative to the 38 hours required in the logic model. 
The index was computed by school, since each school had one interventionist. The evaluators 
established a score of 1.00 or above as adequate fidelity to the professional development model, 
while scores below 1.00 were considered inadequate. 


The ISBE Project Coordinator, LEA Project Coordinators, and school principals were not 
required to attend the professional development activities. Considering the focused nature of the 
intervention, participation in professional development for non-teaching staff was not considered 
a part of the fidelity of professional development model. 


Classroom model 


The evaluators worked closely with Cambium’s research department to ensure that the site visit 
rubric reflected the conceptual framework and format of PRJ II. The rubric included four 
components. The first component, Section A: classroom environment, provided a descriptive 
overview of classroom size, desk arrangements, technology elements, and materials required for 
the intervention. Although not directly related to the IFI, the elements in Section A reflected the 
evaluators’ experience with implementation of educational programs. Sections B through D 
reflected the IFI. Section B and Section C focused on the quality and amount of instruction and 
use of differentiation strategies. Section B provided an overview of the lesson’s structure, while 
Section C was lesson-specific, thus the template changed according to the lesson number within 
the Expedition. Section D had elements of the classroom management component found in the 
IFI. Since the observers would not be present during assessment time, all information from the 
assessments was obtained through VPORT. Table 5 displays the alignment between Cambium’s 
IFI and the rubric developed for the site visits (see Appendix B for a copy of the rubric). 


Table 5. Alignment between IFI and evaluator observation rubric 
Voyager Index of Fidelity of 
Implementation (IFT) 


RMC Classroom Observation Rubric 


A. Classroom environment 


Quality of instruction 


B. Lesson planning and delivery — Overview 


Amount of instruction : ; is 
C. Lesson planning and delivery - Lesson-specific 


Differentiation 


Classroom management D. Classroom behavior/management 
Use of assessments Obtained through VPORT 
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During each school visit, two evaluators observed the interventionist during an entire class 
period and each observer completed the observation rubric independently. Following each 
observation, the evaluators met to discuss their ratings. After the first round of observations 
(approximately 5 classroom observations), the evaluators met to discuss any needed changes to 
the observation rubric and to address questions regarding the observation process and ratings. 
The discrepancies in observer ratings were minimal and did not indicate any problems with the 
overall scoring process. 


The process to calculate the fidelity score for the classroom model was as follows: (1) all 
observation rubrics completed by every member of the evaluation team were entered into the 
observation database; (2) the different observations for each evaluation team were combined to 
get an average score across all observers for each lesson; and (3) the scores for both rounds of 
observations were then combined to get an average score for each interventionist. Because each 
school had only one teacher, the teacher score equals the school score. Table 6 displays the final 
model for calculating the index of fidelity of classroom implementation. Based on feedback 
from Cambium, the evaluators established the following fidelity levels: scores below 0.70 were 
defined as inadequate or low implementation, scores between 0.70 and 0.89 reflected medium 
fidelity; scores of 0.90 or above were considered high fidelity. 


Table 6. Calculating the classroom implementation fidelity score 
; Total Possible 
Section Score Weighted Score 
X/6 
Xp/(12 — number of N/A) 
Xc/8 
Xp/(total time intervals — number of N/A) 


Total possible score 
Levels: 0.0 — 0.69 = low 0.70 — 0.89 medium 0.90 — 1.0 = high 


It is important to note that the fidelity of implementation score was not used in the impact study. 
Since the score had solely a descriptive purpose, the evaluators decided to keep the two scores 
(professional development model and classroom model) independently, rather than creating one 
score that would be less descriptive. The next section presents the outcomes of the first year of 
ISR project implementation. 


Implementation Year 1 (2010-2011) 
Context of RPJ implementation 
Reading Implementation Teachers 
Six full time RITs were hired by the ISR project, one per school, following the procedures 


established at the outset (Part I, ISR Logic Model). Each RIT taught 3 to 5 PRJ III classes daily 
and all remained with their classes for the duration of the school year. 
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Classroom space 


Table 7, on the next page, presents the student enrollment information for the different schools 
and grade levels for the beginning of the SY 2010-2011, as collected in VPORT. During the first 
implementation year, PRJ III classes varied in size from 7 to 16 students, with an average of 12.7 
students per class. This information reflects the number of students that were actually entered 
into the system as attending the PRJ IIJ classes, and not the number of students who took the 
outcome assessments. The input of students’ names and results of assessments into VPORT was 
the RTIs’ responsibility. Considering that 427 students had been randomly assigned to the 
treatment group during the summer of 2010, it appears that 29 percent of the assigned students 
never started the intervention. Information obtained on these students indicated that they either 
left the schools before classes began or were retained in eighth grade. 


Table 7. Student Enrollment in Passport Reading Journeys LIT 
School Total Students Periods taught Average Students/ Class 


Source: VPORT enrollment records 


All teachers had “permanent” classrooms, that is, they had regularly scheduled space to conduct 
the intervention. During the site visits, the evaluators deemed that only one school had a class 
that was too small to accommodate small group instruction. All other classrooms were deemed 
adequate in terms of space for instruction. 


Professional development model implementation 
Required professional development 


The two VIS assigned to the project provided the RITs with eight hours of individualized launch 
training onsite before the beginning of the school year. The training focused on how to 
implement the intervention with fidelity, the assessment system, and classroom management. 
The VIS conducted a second day of individualized professional development in January 2011 to 
clarify questions. The RITs also had available to them 16 hours of online modules in VPORT 
for further clarification of the topics covered in the launch training. These hours were optional. 
Additionally, the RITs were required to attend the IRA annual convention in May 2011. Table 8 
displays the hours of professional development completed for each RIT. As the table indicates, 
all RITs completed their hours of required professional development, and five RITs completed 
different portions of the online modules to exceed the required hours. 
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Table 8. Hours of professional development 


Aaya Schools 


Launch training 


Data meeting 

Online PD (optional) 
IRA Convention 
Total PD Hours 


Individual Supports 


The VIS were contracted to provide a maximum of 25 hours of on-site coaching to each RITs (5 
visits per teacher; 5 hours per visit). All RITs reported receiving visits from the VIS and 
highlighted three topics as the most frequently discussed during the coaching sessions: analyzing 
data, sharing resources, and modeling instruction. During the monthly check-ins, the RITs also 
reported maintaining regular attendance at the conference calls with the ISBE Project Director. 
They mentioned the articles they had to read for the call and the support received from the 
Project Director and from each other. 


Table 9 includes information related to the individualized supports received under the ISR 
project. As seen in the table, the RITs indicated receiving more hours of individualized support 
from both the VIS and the ISBE Project Director than initially planned. Only one RIT indicated 
having fewer support hours than planned. Despite attempts to triangulate the information from 
the RITs with that from the support providers, conflicts remained regarding the amount of 
individualized supports. A partial explanation for the discrepancies was that some RITs included 
in their report the supports provided via e-mail or telephone, while others reported only the 
supports provided in person or by conference call. 


Table 9. Hours of individualized supports 


ae Schools 
Activities 1 


Months reported 


Expected hours coaching and supports 


Actual hours VIS coaching 


Hours of coaching from ISBE Project Director 


Total hours of individualized supports 


Index of fidelity of implementation: professional development model 


Based on the required professional development hours, all interventionists received an index of 
1.0 or above, which was defined as the adequate implementation level for the IRS professional 
development model. Across schools, the index of fidelity of professional development ranged 
from 1.0 (school 5) to 1.6 (school 6). Table 10, on the next page, summarizes the information on 
the professional development and individualized supports provided to the RITs during the first 
implementation year. 
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During the monthly check-ins, the RITs were also asked to rate the individual supports received 
in three characteristics: relevance, sufficiency, and alignment between the two support providers. 
RITs were asked to use a rating scale from 1 (irrelevant/insufficient/ not aligned) to 5 (highly 
relevant/sufficient/aligned). The highest average score across schools was for the alignment 
between the supports received from VIS and ISBE Project director (4.34); the average score for 
the two other scales (relevance and sufficiency) was 4.24 (relevance) and 4.21 (sufficient). 

Table 10 summarizes scores per school. 


Table 10. Index of fidelity of implementation — professional development model 


hool 


Hours individualized | VIS 
supports! ISBE Project Director 
Hours of PD and Convention attendance 


Total hours PD and supports 


Index Fidelity of implementation - PD model?” 1.24 


Relevance 


Individualized 


supports (perception) Sufficiency 


Alignment 


Not included in the index calculation 
Required PD = 38 hours 


Implementation of the classroom model 


As previously discussed, PRJ III is formatted as a series of lessons, or Expeditions, each 
comprised of ten lessons. The RITs were expected to cover one lesson during a 50-minute 
period or two shortened lessons if using 90-minute block scheduling. Lessons 5 and 10 of each 
Expedition, used to re-teach or expand the information provided in the previous lessons, could 
take two days. Each Expedition was thus intended to be completed in 12 days. Cambium 
provided suggestions on how to modify the lessons to fit the classroom period. 


The evaluators conducted two visits to each of the participant schools to monitor classroom 
instruction for a total of 27 observations. The first visit occurred in October 2010, at the end of 
the first month of implementation, and the second visit occurred in February 2011, as the 
implementation entered its sixth month. Two other visits were scheduled — one for mid-March 
and the other for the end of May — but the visits were cancelled to address cut in grant funds. 
Data from the observations were analyzed with two foci: intensity of the intervention and fidelity 
to the developers’ model. 


Actual intensity of intervention 
The actual intensity of the intervention the students received was influenced by three elements: 


(a) actual instruction time; (b) actual days of instruction; and (c) student attendance (or the need 
to re-teach for absent students). 
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Actual versus allotted classroom time: When considering instruction time, two measures must be 
considered — allotted time vs. actual time in instruction. All six schools used a 90-minute period 
for the intervention. However, classes that happened during the first period tended to be 
curtailed by announcements, and those in the last period could be shortened for early class 
dismissal, assemblies, or meetings. Interruptions due to student behavior would further deplete 
from the allotted instruction time and may explain the difficultly that some RITs had in complete 
one lesson per day. Table 11 displays information on the number of lessons observed during the 
visits and the number of lessons completed during the observation. During the two site visits, the 
evaluators were able to observe four lessons in schools 1, 3 and 4; five lessons in school 5; and 
six lessons in school 2. Two of the RITs were able to complete the expected number of lessons 
during the observation, while the others had varied success in completing their lessons. 


Table 11. Lessons completed during class time 


Honea Schools 
| : 


Number of lessons observed 4 4 

Lesson completed in class 2 3 
Days of instruction: During the monthly check-ins, the evaluators asked interventionists to report 
the number of days the school was closed as well as the number of classes that were cancelled 


during the time period covered by the check-in. Table 12 presents information on school 
closures and class cancellations as reported by the teachers. 


Table 12. School closures and class cancellations 


hool 
Number of Days Schools 


School closures 


Cancelled classes 


Total missed classes 


The list of closures included holidays, teacher working days, and inclement weather. Reasons 
for class cancellations included professional development days, testing, and Positive Behavior 
Interventions and Supports (PBIS) award ceremonies. Overall, in the nine-month period of 
implementation data collection, students missed close to one month of instruction. It is 
important to observe that, although school closures appear to be high, many of those days were 
already part of the regular school calendar.* More importantly, the numbers of class 
cancellations were quite low, varying from 0 to 9 within September 2010 through May 2011. 


Student attendance: Students’ presence in the classroom was the final element considered to 
influence the dosage of the intervention. The evaluators had initially planned to request student 
attendance records from the participating schools at the end of the school year. Once the 
Striving Readers grant funding was cancelled, the evaluators chose not to place this additional 
burden on the schools and to focus resources on obtaining the student achievement data. 
Nevertheless, during the monthly check-ins, the evaluators asked interventionists to report on 


* Tlinois require a minimum of 176 days of teacher/student contact; school districts can exceed these required days. 
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student attendance during the month covered by the check-in. Table 13 displays the percentage 
of “yes” responses to the question of “almost all students attended” PRJ //I instruction within the 
month in check. For school 1, for instance, the RIT responded “yes” to the first option in 5 out 
of 8 check-in times. The main reason for sporadic attendance was student behavior. For 
instance, within the period of a month, one of the RITs had seven students suspended for a total 
of 20 days of missed classes, while another RIT had suspensions for a total of 34 missing days. 


Table 13. Student reported attendance 
a 


Responses (%) 


Almost all students attended PRJ 


Expeditions completed: The actual intensity of the intervention may be assessed by the number 
of Expeditions completed during the school year. Cambium establishes a pacing calendar for the 
interventionists to ensure that all Expeditions are covered within the school year. However, as 
discussed above, a number of factors intervened with the planned calendar. Table 14 displays the 
percentage of check-ins when the interventionists reported being on or off schedule regarding the 
pacing calendar. 


Table 14. Pacing of instruction 


hool 
Responses (%)' 7 echools 


On schedule 
Ahead of schedule 
Behind schedule 


Expedition completed? 


Percent of responses during the check-ins 
? Information retrieved from VPORT on November 2011 


As seen in the table, the majority of time, the interventionists were behind the schedule. VPORT 
includes information on the assessments conducted in the final lesson of each Expedition (lesson 
10) with the number of Expedition. This information is displayed in the last row of the table. Of 
the 14 Expeditions that form the PRJ I/I curriculum, completion rates varied from 29 percent (4 
Expeditions) to 78 percent (11 (Expeditions). However, the information must be viewed with 
caution, since it refers to assessments administered and entered into the system by the RITs. If 
the RIT was not up-to-date with the data input into the system, the information will be 
misleading. 


All RITs had sufficient allotted class time to complete a full PRJ lesson per day, according to 
Cambium’s parameters (50 minutes daily or 90 minutes every other day). However, actual 
instruction time varied from allotted time and classes could be curtailed for morning 
announcements or early dismissals. Additionally, classes could be cancelled or students would 
be missing for days, two events that disrupted the RITs’ pacing. In fact, during the check-ins 
with interventionists, the issue of pacing was one of the most frequently cited concerns. Also, 
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RITs were concerned about striking a balance between maintaining the brisk pace while ensuring 
that all students were learning. 


During the interviews, the VIS confirmed that pacing was a major issue. “How do I get through 
this material in the amount of time and cover it with fidelity?” was a frequently heard question, 
according to one of the two VIS assigned to the project. To address the RITs’ concerns, the VISs 
suggested strategies to keep the lesson moving, while making sure that students were learning, 
and using the extra days in lessons 5 and 10 to maximize their efforts. Another recommendation 
was to let the students who scored on the higher Lexiles do more independent and challenging 
work to keep them engaged, while leaving the RITs with more class time to concentrate on the 
students with the lower Lexiles. The two VIS were discussing the possibility of extending 
lessons 5 and 10 to three days to allow for more re-teaching or catching-up, a change that was to 
be instituted the following implementation year. 


Actual student use of technology 


The ISR budget included money for the purchase of hardware to ensure that all classrooms had 
the needed equipment for the intervention, specifically computers and DVD projectors. The 
main technology component of the PRJ III is the VocabJourney, an interactive online game 
designed to help students with vocabulary practice tailored to their individual needs. Students 
could access VocabJourney at any time from any computer. The RITs reported that some 
students were very excited by the game and played it quite frequently (even at home) in order to 
earn as many points as possible. Alternatively, other students appeared totally disinterested in 
the game. 


During the site visits, the evaluators noted that interventionists often asked students to play 
VocabJourney during any “down time” in instruction; for example when finished with an 
assessment or waiting for other students to complete their work. It is of note that Cambium does 
not recommend this practice. Instead, Cambium recommends that the teacher monitors student 
learning on VocabJourney and immediately intervene with individualized support as needed. 
The observers also noted, and interventionists reported, multiple problems with accessing 
VocabJourney during class time. RITs mentioned frequent computer “freezes,” difficulty 
accessing the online portal, and problems with internet connectivity, among others. They also 
reported difficulty with the DVDs provided with the PRJ. Evaluators observed them struggling 
to find the correct location on the DVD that corresponded with a particular lesson or having 
problems with the DVD not loading correctly. Cambium was responsive to interventionists’ 
requests for assistance, and it is the evaluators’ understanding that Cambium even made some 
changes to the DVDs based on the RITs’ feedback. RITs in all four school districts were using 
interactive whiteboards to provide more opportunities for students to actively participate in the 
lessons and Cambium experts were adapting lessons for use on the interactive platforms. 


Student assessment procedures actually carried out 
As detailed in Part I, ISR Logic Model, the PRJ III assessment system includes measures of text 


fluency and comprehension, end-of-lesson assessments, progress monitoring, and student self- 
assessments. The assessments were administered as planned and each RIT had a calendar with 
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the dates or times for the three Reading Benchmark tests, the Semester assessments, and the 
ongoing comprehension and vocabulary tests. Students’ performances in the different 
assessments were recorded on VPORT and RITs used the information to individualize 
instruction according to information provided in the monthly check-ins. The school divisions 
also administered the GMRT® and Explore® assessments used for the impact study, as discussed 
in Part ITI. 


VPORT displays results from the PRJ III assessments by school, class, and individual students. 
Table 15 displays a summary of class results on the three benchmark assessments administered 
during SY 2010-2011 as an example of the information provided. In the table, the results are 
displayed in Lexiles. 


Table 15. Example of reporting on the benchmark assessments 


Scores (Lexiles)! 
B2 B3 


Lexiles measured from OL — 1500L 
? Gain= B3-B1 


Calculating fidelity of implementation 


The classroom observation rubric comprised four sections. Sections A (context) and B (general 
lesson plan and delivery) were scored on a two-point scale whereby scores of 2 reflected 
adequacy, while scores below 2 reflected inadequacy. Section C (specific lesson plan and 
delivery) used a three-point score for C1 and C2 and two-point score for C3. Section D 
(classroom behavior and management) reflected percent of time where the specific behaviors 
were observed during the evaluators’ visits. Each section received a weight that addressed the 
importance of the section within the overall classroom implementation model, as adapted from 
Cambium’s IFT. 


Table 16, on the next page, displays the scoring for each component in the four sections of the 
rubric. Based solely on the adequacy of delivery observed during the site visits, two of the six 
interventionists would have been classified as attaining high fidelity of implementation (score at 
or above 0.90), three would attain medium fidelity (scores between 0.70 and 0.89, and one would 
be classified as low fidelity (score below 0.70). 
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Table 16: Scoring for the classroom observation rubric 


hi “ 
Number of classes observed 
Classroom time observed (minutes) : : 


Section A: Classroom Environment 


Al - Sufficient space : 1.50 
A2 - Instructional areas : 1.50 


A3 - Teacher resources é 1.80 
A4 - Student materials ‘ 2.00 
Section score: X,/8 ‘ 0.85 


B1 - Follows curriculum guide 
B2 - Brisk pace 

B3 - Skills modeled 

B4 - Correction procedures 


BS - Students in groups 
B6 - Differentiation 
Section score: Xp/(12 — number of n/o) 


C1 — Components delivered in order 


C2 — Steps delivered in order 


C3 - Completed within suggested timeframe 


Section score: X-/(6-number of n/o) 


Section D: Classroom Management 
D1 - % time students pay attention 1.00 | 1.00 
D2 - % time students respond to prompts 0.80 | 0.70 


D3 - % time students actively participate 1.00 | 1.00 
D6 - % time students follow expectations 0.90 | 1.00 
D4 - % time teacher addresses behavior (x2) 1.60 | 1.70 
DS - % time teacher engaging students (x2) 2.00 | 1.10 
Section score: Xp/(total time intervals —N/A) 0.90 | 0.81 

Weighted scores 


Sections Weight 


Classroom Fidelity Index 
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Table 17 displays the scores for the fidelity of implementation of both models (professional 
development and classroom implementation). According to these indices, all schools attained 
adequate implementation of the professional development model, while the fidelity of the 
classroom model ranged from inadequate to high fidelity. 


Table 17. Fidelity scores per school 


Classroom 


PD 
Classroom 


Legend: I= Inadequate; M = Medium (Adequate); H= High; 


Factors influencing the implementation 


The initial framework which was proposed for the analysis of the PRJI/I implementation 
included the two components of the logic model — professional development model and 
classroom instruction model. A third component was added to include elements that was not 
directly connected to the reading supplemental program, but which the evaluators considered 
relevant from their experience with the evaluation of education programs and initiatives. This 
component, called context, comprised two elements: physical resources and time in instruction, 
which included actual length of daily instruction (allocated time — time used for activities 
unrelated to instruction, such as announcements or classroom management) and days dedicated 
to instruction. 


Two hypotheses were discussed on how to incorporate this third component into the framework. 
The first hypothesis considered context as a moderator of the classroom implementation, 
strengthening or weakening the implementation of the classroom instruction model. The second 
hypothesis considered context as an independent component, with similar weight as PD and 
classroom instruction. A provisional decision was made to incorporate context into the 
classroom instruction model for year one and, using a grounded theory approach (Glaser & 
Strauss, 1967), “ask questions” to the first year data in order to fine-tune the model and the data 
collection process for year two. 


During the planning year, the implementers made an effort to promote consistency across the 
schools. Teachers had similar background in terms of licensure and reading endorsements and 
received similar general professional development and individualized supports that addressed 
their particular needs, while the ISBE Project Director worked closely with the district liaisons to 
support the implementation within each school. Yet, levels of fidelity of the classroom 
implementation varied across schools. One school received a below adequate score, three scored 
within the medium implementation range, and two attained high level of implementation scores. 
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The analysis of implementation data revealed two main factors that worked as general 
contributors, either facilitating or hindering the implementation in all the schools, and four 
factors that varied across schools. A discussion of these factors follows. 


General factors 


Professional development and supports: Within the two components of the logic model, 
professional development and supports seemed to have worked as a “general” factor that 
facilitated the fidelity of PRJ/I implementation at all participating schools. During the 
interviews, the VIS commented on the overall supports they and the RITs were receiving from 
the ISBE Project Director, the LEAs Project Coordinators, and the school principals. “Principals 
have been very supportive, reshuffling classes to minimize behavior problems, and rearranging 
classrooms to make them more welcoming,” observed a VIS. The LEA coordinators were 
described as very supportive and involved in the process, acting on feedbacks from the VIS and 
helping the RITs with needed resources. VIS and RITs alike commented on the benefits of the 
community of practice (CoP) organized by the ISBE Project Director. Regarding the CoP, the 
RITs particularly appreciated the opportunity to have a forum where they could exchange 
experiences and ideas and support each other. 


Intervention format: For the second component of the logic model, classroom instruction, 
findings were more complex. The format of the program seemed to have worked as a “general” 
challenging factor for the implementation. Pacing and routine were the two characteristics of the 
intervention that received the most comments during the RITs’ check-ins and site visits. As 
described in Part I, Description of the intervention model, PRJ III blends a repetitive approach of 
tightly structured daily lesson components with some flexibility to address the needs of an older 
student population. This includes more independent and paired student work than in the PRJ 
programs for lower grades (Beginnings, I and II). To allow for flexibility, Cambium does not 
establish the timeframes for each daily lesson component in PRJ III, as it does with the lower 
grade PRJ programs. However, in the interest of promoting implementation consistency across 
sites for the ISR project, Cambium, in consultation with the ISBE Project Director, developed 
suggested timeframes for each daily lesson component. 


Interview and observation data indicate that RITs were struggling (with different levels of 
success) to attain the balance between the apparently conflicting requirements of keeping a brisk 
pace and completing the required lesson components in sequence while providing students with 
some flexibility and independence. The RITs also observed that students complained about the 
“sameness” of the lesson formats and attributed some of the observed student behavior to 
boredom. During the check-ins, the RITs recognized that Cambium was making changes in the 
program to address their concerns. However, as a VIS commented, establishing the routine was 
essential. When the implementation progressed successfully, commented the VIS, “you can tell 
that a routine is established, because the [students] know what to expect.” 


Specific factors 


The analysis of the first year data suggests that neither of the two above factors influenced the 
implementation process by itself, but it was through a complex interaction with four elements 
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that are being called here “specific factors.” For instance, although all RITs received the same 
type and amount of professional development related to the program, some showed greater ease 
in handling the program’s challenges, including pacing and routine, while others struggled with 
them. Likewise, although the program format was the same across school, some RITs were 
dealing with issues of space that were not common to all schools and that appeared to have 
influenced how the program was implemented. The specific factors highlighted in the analysis 
of the first implementation year included: classroom management, defined as the teacher’s 
ability to maintain student engagement throughout class time; classroom space; actual time in 
instruction; and implementation time. These elements appeared to behave not as factors in 
themselves, but as moderators of the two major components of the implementation. The role of 
these four elements would merit further exploration in year two if the grant had continued. 


Classroom management: During the site visits, the evaluators observed a number of students 
disengaged from instruction and the RITs’ sometimes unsuccessful efforts to engage them. It is 
of note that disruptive behavior was not commonly observed during the site visits, and the RITs 
or VIS did not comment that such behavior was a concern. The evaluators observed mostly off- 
task chattering, occasional “back talk,” and students not responding to teachers’ questions. As 
commented above, student “boredom” of the program routine was suggested by the RITs as a 
major reason for this off-task behavior. However, it was also clear that response to the 
“boredom” was different across the schools. Some RITs resorted to creative strategies to 
motivate students and increase participation, such as having students toss a ball to one another to 
respond to questions. Another said, “J ended up typing the discussion questions ahead of time to 
give to students, because it’s hard to get them to discuss in the group; they just don’t feel like 
talking.” Other RITs were less successful in their attempts to engage their students in the tasks 
at hand, which resulted in incomplete student assignments and/or inability to gauge student 
understanding of content taught. Likewise, although pacing was a common challenge across all 
schools, the ISBE Project Director commented that, “the more effective teachers in the group are 
figuring out how to deal” [with pacing], while others were clearly struggling and unable to finish 
their lessons within the allotted time, as detailed in Table 14 (p. 36). A major part of this 
“figuring out” was related to the teachers’ ability to keep students on-task and minimize 
disruption to instruction. It is important to highlight that classroom management should not be 
interpreted as teachers’ sole responsibility or ability. As discussed above (professional 
development and supports), school administrators had an important role in facilitating classroom 
management by “reshuffling classes” to minimize disruptive student behavior and rearranging 
classrooms to make them more welcoming. 


Classroom space: Within context, classroom space was one component that differed at least in 
one of the schools.’ It is of note that the only school where classroom space was a challenge 
also scored low on other measures, including overall lesson plan and classroom management, 
and the overall fidelity level was below adequate. The actual impact of space (or lack thereof) 
on instruction was not clear, particularly as the classroom observations were cut short due to the 
termination of the grant. Within the limited available data, lack of space was seen as influencing 
the teacher’s ability to organize the room for effective group work, an integral part of the lesson 


* Asa side note, inadequate space as a barrier to fidelity of implementation was also a finding in another Striving 
Readers project evaluated by RMC. 
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plan, and to maintain students engaged in the lesson, as the physical proximity between students 
facilitated off-task conversation. 


Actual time in instruction: Also under context, actual time in instruction was another factor that 
influenced the implementation of the classroom model. Time in instruction is here defined as 
allotted class time (minutes allocated in each period) minus time dedicated to activities unrelated 
to instruction, such as morning announcements, administrative tasks, or time spent repeating 
lessons for chronically absent students. Time off instruction due to other school-related 
activities affected mostly the first period classes through the morning announcements. Yet, RITs 
did not appear affected by this element, and most of the comments during check-ins were related 
to the “chronically absent students.” 


Students were absent either for being suspended or simply avoiding attending the class. This 
reflected not only on missing days (that is, dosage of instruction), but also on time in instruction 
as teachers had to repeat information previously covered when the absent students returned to 
class. The study format was the reason for this factor. RITs commented that some students had 
been assigned to the treatment due to a low score in the grade 8 assessments (the eligibility 
criterion), but scored high in the PRJ I/I benchmark assessments. These were probably good 
readers who simply did not take tests “too seriously,” explained one RIT. The perception from 
the teachers was that the students were resistant to the intervention, because they did not need it 
and resented being labeled struggling readers. They simply would not come to class. However, 
the RITs were constrained to keep the students in class to maintain the fidelity of the random 
assignment. The data did not clarify how many schools were affected by this conflict between 
low grade 8 EXPLORE® score but high PRJ/II benchmark score. At least one of the schools 
showed a clear impact of these students in the overall level of implementation. Had the study 
continued a second year, the data collection protocol would be changed to identify the number of 
students under this category and their attendance patterns. 


Implementation time: Interviews with teachers and VIS suggested that the influence of 
classroom management and program format on classroom implementation was inversely related 
to familiarity with the program. As teachers progressed with the PRJII implementation, and 
started feeling more comfortable with the program, they felt that student behavior and, 
concomitantly, classroom management became less of a problem. Likewise, pacing became less 
of a challenge. A comment from one RIT summarizes the impact of time on implementation 
familiarity, 


Since October, discipline is a lot better. [The students] are maturing a bit. A bit more 
understanding of why we're here doing this. They know the routine so we can do it 
faster. We're able to do some writing now. Now I know we can make some 
modifications. 
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Factors influencing the impact analysis 


Regarding the impact analysis, two factors should be highlighted. The first factor was the 
variation in implementation across the schools, which resulted in a score of middle 
implementation for the overall project. The second factor, highlighted by the RITs, was that the 
outcome assessments were administered before the teachers had the opportunity to cover the 
curriculum. Students were still making progress but had already been assessed for impact. It is 
not clear how much another month of class would have changed the results of the reading 
assessments. Yet, since none of the teachers were able to finish the “regime,” using a medical 
metaphor (none went beyond Expedition 11), studies where the “regime” is allowed to be 
completed may bring a better understanding of the impact of a supplemental program on 
adolescent struggling readers. 
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PART Il: IMPACT STUDY 


Study Design 


The evaluation design proposed by RMC Research Corporation (RMC) to investigate the impact 
of the Illinois Striving Readers (ISR) project on student academic performance was to be guided 
by three major research questions: one exploratory and two non-experimental questions. The 
main (exploratory) question pertains to treatment (PRJ II) effects directly targeted with the 
experimental design of the JSR. The two non-experimental questions were intended to capture 
reading achievement of students before, during, and after their experimental exposure to PRJ III. 
With the abrupt cancellation of the study, only the main impact question was addressed. 


¢ RQ1 [Main Impact Question]: What is the experimental (pretest-posttest) effect of the 
PRJ III supplemental literacy intervention on reading achievement of grade 9 students at 
the end of the first implementation year (SY 2010/11)? 


RQI1 is addressed by using a two-level Hierarchical Lineal Modeling (HLM) model that takes 

into account the multilevel structure of the data defined by nesting students within schools. The 
Gates-MacGinitie Reading Tests® (GMRT) 4" Edition and the reading component for the Grade 
9 EXPLORE® were used to measure the reading achievement of treatment and control students. 


Sample selection 


As described in the /ntroduction, six high schools in four school districts in Illinois participated 
in the study. All of the schools were eligible to receive funds under Title I, Part A, pursuant to 
ESEA section 1113. The percentage of low-income students in these schools during SY 2008-09 
ranged from 44 to 66 percent and the grade 9 enrollment ranged between 390 and 587 students 
per school. 


Students enrolled in grade 9 at the six participating high schools who had records of taking the 
grade 8 EXPLORE? in the spring were considered for Striving Readers eligibility. Data on 
results from the ACT’s assessment for the first project year were provided to the evaluator by the 
LEA Project Coordinator from the four participating districts during May and June 2010. This 
early date was important for the schools to ensure a smooth transition of students from grades 8 
to 9 and allow for careful scheduling of all the students. 


The total number of incoming ninth graders across the six schools was 1,985 in the first program 
year. Students who scored above 12 on the reading component of the EXPLORE® were 
determined to be ineligible for the study. The cutoff score of 12 was determined based on the 
following criteria: 


e The cutoff score of 12 is approximately the oT percentile of students, which is the 
recommended cutoff for the low performance category with normally distributed 
assessment data (Kelley, 1939). Preliminary data analyses revealed that the distribution 
of EXPLORE® reading data was normal. 
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e This cutoff score allowed for obtaining an adequate sample of eligible students that 
satisfies the minimum number of student participants required by the USED. 

e The cutoff score matched previous school practices and teacher experiences in 
identifying struggling readers. 


All students scoring at or below this criterion score were included in the list of eligible students 
returned to the schools so they could apply exclusions before the random assignment. The 
exclusions included students who had an Individualized Education Program (IEP) specifically 
forbidding their participation in a program like Striving Readers, and students whose parents 
requested that they be excluded from the study. Additionally, students who were retained in 
grade 8 were also excluded from the list. 


Documenting attrition 


After students were randomly assigned to the treatment and control groups, students who 
transferred to another participating school during the school year would continue their 
assignment as treatment or control group students. Only one school district had a high school 
that was not involved in the project. Therefore, for this LEA, data would be requested for the 
students who transferred to a non-participant school, as they would continue to be part of the 
impact analysis based on Grade 9 -EXPLORE® testing results but not on GMRT® results 
(GMRT® is not administered to students in schools outside of the Striving Readers project). The 
only students expected to be missing from the impact analyses would be those who missed the 
spring administration of the two outcome measures and those who transferred or otherwise 
stopped attending schools in the participating school districts. ISBE had indicated that a 10 
percent annual attrition rate was to be expected. As a precaution, planning was made to study 
differential attrition by determining if there was a significant interaction between attrition status 
and experimental condition on baseline outcome measures. 


Random assignment 


The random assignment of students to intervention and control groups was conducted after 
obtaining the lists of eligible students with exclusions applied from the schools. A total of 855 
students were included in this final list. The process for the random assignment included three 
steps. First the eligible students were assigned to pairs, with the students in each pair matched 
on relevant characteristics: EXPLORE” reading score, Limited English Proficiency (LEP), free 
and reduced meals (FARM) eligibility, special education status (SPED), gender, and ethnicity. 
Students were matched on as many criteria as possible. Then the students from each pair were 
randomly assigned to two different groups using a Bernoulli distribution function with p = 0.5. 
The process was completed by randomly assigning the two groups to the two treatment 
conditions (intervention or control). The resulting intent-to-treat ITT) group, comprised of 855 
students, represented 43 percent of the total enrollment across the six schools. 


As soon as the random assignment was conducted, the evaluators provided the school 
representatives with the lists of students assigned to treatment and control group. Open 
communication between the evaluators, the LEAs, and the schools ensured that the transfer of 
these lists was efficiently handled and questions and concerns were addressed. The random 
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assignment results were provided to the districts between 5/10/10 and 6/30/10, depending on 
when the reviewed eligible lists were returned to the evaluator. Each school scheduler initiated 
the process of assigning students to their respective classes, with treatment students attending 
PRJ III while control students were to enroll in electives unrelated to reading supplemental 
intervention. Table 18 presents ITT group breakdowns by school. 


Table 18. ITT breakdowns by school 


School Treatment Control Hee Total 
District stew Group Group Non-Eligible Enrollment 


Danville Danville 


Eisenhower 
MacArthur 
Lanphier 


Decatur 


Springfield 3 
Springfield Southeast 


Kankakee Kankakee 
Total 


Demographics of ITT group 


The demographic makeup of the ITT group was primarily African American (58%). White 
students constitute 30 percent of the ITT group, while other races (American Indian, Asian, 
Hispanic, mixed, and other) represent 12 percent. The ITT group contained slightly more males 
than females, and the majority of students (85%) qualified for free and reduced lunch, which is 
used as the proxy for low socioeconomic status. Table 19 displays the demographic breakdowns 
for the ITT group. The matching procedure used in the random assignment process ensured that 
the demographic percentages were equally distributed between the treatment and control groups. 


Table 19. Demographic breakdown of ITT group 


Subgroups Percentage 


African American 


Hispanic 
White 
Other 


Race/ Ethnicity 


Male 56.7 
Female 43.3 
Free and Reduced Lunch 84.7 
Special Education 17.6 
Total (N) 855 


Gender 


Random assignment monitoring 


To monitor the integrity of the random assignment process the evaluator accessed VPORT, 
which allowed for inspection of which students were receiving the treatment. Also, the evaluator 
exchanged information with LEAs to confirm the status of ITT students assigned to control. The 
first monitoring review took place in December 2010. Information on attrition was gathered 
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from the LEAs to document the reasons why students were missing from their respective ITT 
group. Common reasons were: exiting the school; not being in the ninth grade (that is, retained 
in grade 8 after the random assignment had been completed); excluded after the random 
assignment process due to special education requirements; and not having available test data. 
Figure 3 displays the consort diagram showing the ITT group attrition in regards to EXPLORE, °® 
while Figure 4, on the next page, displays the attrition in regards to GMRT’. 


Figure 3. Consort diagram — ITT group attrition in regards to EXPLORE® 


POPULATION OF STUDENTS 
N= 1985 


ELIGIBLE STUDENTS INELIGIBLE STUDENTS 
n= 855 n= 1130 


RANDOM ASSIGNMENT 


TREATMENT GROUP CONTROL GROUP 


n= 427 n= 428 


f f 


Not Eligible for Study Post-RA Eligible for Treatment Eligible for Control Not Eligible for Study Post-RA 
n= 1 n= 427 n= 428) n= 2 


Attrition Post-RA Attrition Post-RA 
n= 19 n= 10 
Sample at Baseline Sample at Baseline 


407 416 


Attrition at Follow-up Final Intent-to-Treat Final Intent-to-Treat Attrition at Follow-up 
n= 143 Analytic Sample at Analytic Sample at n= 166 
Attrition due to: Follow-up Follow-up Attrition due to: 
*Extied School *Extied School 
*Notin 9th grade *Not in 9th grade 


*Excluded post-RA *Excluded post-RA 
*No test data *No test data 
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Figure 4. Consort diagram — ITT group attrition in regards to GMRT° 


POPULATION OF STUDENTS 
N= 1985 


ELIGIBLE STUDENTS INELIGIBLE STUDENTS 
n= 855 n= 1130 


RANDOM ASSIGNMENT 


CONTROL GROUP 
n= 428 


Eligible for Treatment Eligible for Control Not Eligible for Study Post-RA 
n= 427 n= 428 n= 


Sample at Baseline Sample at Baseline 


Of the 363 treatment and control students who exited the study after the baseline: 108 (30%) 
exited the school, 15 (4%) were not in grade 9, and 37 (10%) were excluded after the random 
assignment process. The most common reason for attrition was a lack of test data, which 
accounted for 203 of the 363 (56%) of the students who exited the study after the baseline. The 
value for no test data in the “Attrition at Follow-up” box represents the number of students for 
whom test data was not available and the evaluator was not able to verify the cause for these 
missing data specific to the students. School and district staff indicated that these students had 
either exited the system or were absent on the testing day. They also mentioned that after the 
termination of the Striving Readers funds, RITs were reluctant to pursue testing for students with 
missing data, particularly with a testing instrument that was used specifically as an evaluation 
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tool for the grant. Thus, the number of students for whom test data were not available at the end 
of the year was high. 


Final numbers for student attrition are based on the number of ITT students for whom the 
evaluator was able to gather post-test data. Of the 855 ITT students, 460 (53.8%) had data for 
GMRT pre- and post-tests. A dummy variable adjustment procedure was used to account for 
missing data in 17 instances, where the students had a GMRT post-test score from the spring, but 
no GMRT pre-test. For EXPLORE®, of the 855 ITT students, 514 (60.1%) had data for pre- and 
post-tests. A dummy variable adjustment procedure was used to account for missing data in 8 
instances, where the students had an EXPLORE® post-test score from the spring, but no pre-test. 


Data Collection 
The following data were collected for the impact study: 


e Assessment data: 

o Diagnostic (selecting striving readers) — Illinois’ schools use EXPLORE, ® an 
assessment developed by ACT, Inc., with the grade 8 students to assess their 
ability to understand written materials from different school subjects. Results on 
the test were used to compile the list of students eligible for the study, as 
explained above. 

o Impact (assessing performance) — Two assessments were used to evaluate changes 
in student performance between the treatment and the control group students: the 
Gates MacGinitie (GMRT®) an edition, from Riverside Publishing, and the Grade 
9 EXPLORE®, from ACT, Inc. 


e Demographic data: student lists included information on gender, race/ethnicity, limited 
English proficiency, special education status, and eligibility to free and reduced lunch. 
The data were used as covariates in the analysis model. 


Table 20 displays a list of independent variables, the level of analysis where the variables were 
used, when they were to be collected and who provided the variables to the evaluators. 


Table 20. List of independent variables used with the experimental design model 


Variable Provided by Notes on Data 
Treatment Condition 


(ntervention-vs, Control) Student School Fall assignment 


Grade 9 GRMT® [pretest] Student School Annually— Fall (grade 9 only) 


LEP Student School Annually — test administration date 


SPED Student School Annually — test administration date 


FARM eligibility Student School Annually — test administration date 


Gender Student School Annually — test administration date 


Ethnicity Student School Annually — test administration date 
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Data Analysis 


The main impact question— What is the immediate (pretest-posttest) effect of the PRJ III 
supplemental literacy intervention on reading achievement of grade 9 students for the 2010-2011 
implementation year?—was addressed by using a two-level HLM model that takes into account 
the multilevel structure of the data defined by nesting students within schools. 


Level 1 (within-school) part: 


Yi; = Boj + B1j(PRIiy — PRJj) + B2;(PREij — PRE,;) + &;, (1) 
where: 
Yi; = the posttest score of student i in school j on the Grade 9 GMRT®; 
PRJjj;= a treatment indicator variable, with PRJ;; = 1 indicating that student i in school j 
participates in PRJ III, and PRJ;; = 0 otherwise; 
PRE;,; = pretest score of student 7 in school j [on the Grade 9 GMRT"]; 
(PRJi; — PRJ;) and (PRE;; — PRE ;) indicate that PRJ;; and PRE;;, respectively, are 
centered around their school means [group centering, Raudenbush & Bryk, 
2002]; 
Boj = the mean of school j on the Grade 9 GMRT®; (by virtue of the group centering); 
64; = the treatment (PRJ) effect for school j, holding constant pretest performance; 
62; = the pretest/posttest slope for school j, holding constant PRJ (1 or 0); 
€; = errors; €;; ~ N(O, 07). 


Level 2 (between-schools) part: 

Boj = VYoot Yoj; Uoj~ NO, Too), 

Bij =Vi0 + Uy;; Uyj~ NO, 711), 

B2j =Y20t U2;; Uz;~ NO, T22), (2) 
where: 

Yoo = the grand mean [on posttest Grade 9 GMRT® scores]; 

¥10 = the overall average VPRJ effect; 

Y20 = the average pretest/posttest slope; 

To9 = between-school variance of mean scores; 

T11 = between-school variance of VPRJ effects; 

T22 = between-school variance of pretest/posttest slopes. 


Note. A single estimate of the impact of PRJ III on reading outcomes (9 GMRT® scores) is provided by the HLM 
estimate of y,9; (the precision of this estimate is provided by the HLM value of T,,). 


The dependent variable in the impact model defined by equations (1) and (2) is the posttest score 
of students on the Grade 9 GMRT®. Alternatively, the analysis under the same model was 
conducted with the dependent variable being the posttest scores of the students on the Grade 9 
EXPLORE® (the findings from all analyses did not differ across these two dependent variables). 
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Impact on Students at the End of Year One 
Impact on student reading proficiency 


The computations for the estimation of parameters under the HLM model defined by Equations 1 
and 2 were conducted by using the computer program for statistical analysis with latent variables 
in the framework of structural equation modeling Mplus (Muthén & Muthén, 2008). The 
analytic notations used in Equations 1 and 2, however, are consistent with the HLM notations 
adopted in Raudenbush and Bryk (2002) to facilitate the interpretation. The computer program 
Mplus was used for its efficiency in handling HLM missing data under the approach of multiple 
imputations adopted in this study (Allison, 2001; Asparouhov & Muthén, 2010a, 2010b; Muthén 
& Muthén, 2010). Multiple imputations were used to handle missing post-test data, whereas the 
missing pre-test data were replaced using a dummy variable adjustment, as mentioned above. 


The examination of the results in Table 21 shows that the overall PRJ III effect (y19 = 0.39) is not 
statistically significant (p = .720). In addition, the estimate of the between-school variance of the 
PRJ III effects (11; = 3.38) is not statistically significant either (p = .234) thus indicating that the 
GMRT® performance of PRJ III students relative to the comparison group students does not 
depend on school membership. An additional support of this finding is that the estimate of 
overall PRJ III effect under the two-level HLM (y10 = 0.39, p = .720) is practically equal to its 
counterpart obtained under the conventional OLS regression model defined by Equation 3 (8; = 
0.35, p = .666). Note that the OLS regression model is for the entire sample students ignoring 
their school membership. 


Y; = Bo + B PRI; + BoPRE; + §; (3) 
where 
s; ~ N(O, a) is the residual associated with Yi—the posttest score of student i on the Grade 9 
GMRT®. 


Table 21. A two-level Hierarchical Linear analysis under the model defined by equations 1 
and 2 


Fixed Effects Estimate Standard Error p-value 
Grand mean (Yoo) 516.61 1.92 < .001 
Overall PRJ effect (710) 0.39 1.08 .720 


Average within-school pretest/posttest 


0.86 0.03 < .001 

slope (y20) 
Variance Components 

Between school 
ak in site mean on GRMT scores 20.96 12.44 092 

00 

Variance in site PRJ effects (1,;) 3.38 2.84 234 
Variance in pretest/posttest slopes (T2) 0.01 0.01 .167 

Within school 
Residual variance (07) 128.92 14.72 < .001 


Note. The parameter estimates in bold are statistically significant (p < .001). 
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The examination of the results in Table 21 shows also that none of the between-school estimates 
of variance components (To, 711, and T22) is statistically significant. In fact, this result is not a 
surprise given the relative similarity of schools (Title-I eligible schools not making AYP) and 
students scoring at the lowest quartile on the dependent variable measure (ACT’s EXPLORE®). 
Therefore, without danger to validity of the results and their interpretation, we used a 
conventional OLS regression analysis for the total sample of students in all six schools to 
examine the (pretest-posttest) effect of the PRJ II] supplemental literacy intervention on reading 
achievement of grade 9 students for the year 2010-2011. Under the OLS, we also investigated 
for possible effects of the within-school covariates gender, ethnicity, special education status, 
free and reduced lunch, and limited English proficiency using the analytic model in Equation 4. 
Given the dominating sample size for African-American students in all schools (see Table 19, 
p.38), the covariate for ethnicity was designed to examine the effect of African-American 
students versus all other students. 


Y; = Bo + Bi(PRJ;j) + B2(PRE;) + B3(GEN);) + B4(AAj) + Bo(SED)) + Bo(FRL;) + B7(LEP;) + &), (4) 


where 
GEN; stands for gender (GEN; = 0 if female, GEN; = 1 if male), 
AA; stands for African-American (AA; = | if African-American, AA; = 0 if Other), 
SED; stands for special education (SED; = 1 if ‘yes’, SED; = 0 if ‘no’), 
FRL; stands for free and reduced lunch (FRL; = | if ‘yes’, FRL; = 0 if ‘no’), 
LEP; stands for limited English proficiency (LEP; = 1 if ‘yes’, LEP; = if ‘no’), and 
e; ~ N(O, o7) is the residual for Y—the posttest score of student i on the grade 9 GMRT®. 


The results in Table 22 indicate that, except for the regression coefficient of pretest scores, none 
of the regression coefficients associated with the other six covariates is statistically significant. 
As a side note, the same holds when each of these six covariates is taken separately in 
combination of the treatment variable, PRJ (the results are not shown here). 


Table 22. OSL regression for the prediction of posttest scores on the Grade 9 GMRT® 


Variable B SE(B) p-value 
Constant 71.852 10.997 < .001 
Treatment (PRJ) 0.374 0.811 .644 
Pretest 0.869 0.021 < .001 
Gender -0.332 0.824 .687 
African-American -0.104 0.841 .902 
Special education 0.769 1.083 478 
Free and reduced lunch -1.394 1.060 189 
Limited English proficiency -0.533 5.888 928 


The analysis under the HLM model defined by Equations | and 2 was conducted again by using 
the student performance on the posttest Grade 9 EXPLORE® (instead of Grade 9 GMRT®) as the 
dependent variable. The results are summarized in Table 23. The comparison of the results in 
Table 21 and Table 23 shows that the findings described earlier for the HLM model with the 
Grade 9 GMRT® as an outcome variable are identical to their counterparts under the same HLM 
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model, but with the Grade 9 EXPLORE" as the outcome variable. The differences in the 
corresponding parameter estimates in the two tables are due to different scales for the Grade 9 
GMRT® and Grade 9 EXPLORE”, respectively, but this is irrelevant to the (identical) HLM 
findings under these two scenarios. 


Table 23. Two-level Hierarchical Linear analysis under the model defined in equations I and 
2, with the Grade 9 EXPLORE® scores as the outcome variable 


Fixed Effects Estimate SLUTTY p-value 
Error 

Grand mean (Yoo) 12.10 0.14 < .001 
Overall VPRJ effect (y10) -0.26 0.19 168 
Average within-school pretest/posttest slope (720) 0.37 0.07 < .001 
Variance Components 

Between school 
Variance in site mean on ACT scores (tT) 0.08 0.05 .106 
Variance in site VPRJ effects (11;) 0.07 0.07 . 369 
Variance in pretest/posttest slopes (T22) 0.01 0.03 . 952 

Within school 
Residual variance (0”) 5.390 0.845 < .001 


Note. The parameter estimates in bold are statistically significant (p < .001) 


Thus, under Grade 9 EXPLORE® as an outcome variable, the overall PRJ effect is not 
statistically significant ((y10 = -0.26, p = .168) and the variation of the PRJ effects among schools 
is not statistically significant either (t,;; = 0.07, p = .369). Therefore, it was appropriate here 
again to run the conventional OLS regression model defined by Equation 4, with the outcome 
variable being the posttest Grade 9 EXPLORE® scores of the students regardless of their school 
membership that is 


Yj = Bo + Bi(PRJi) + Bo(PRE;) + B3(GEN;) + B4(AAj) + Bs(SED;) + Bo(FRL;)) + B7(LEP;) + &;, (5) 


where Y; is now the posttest Grade 9 EXPLORE® score and PRE; is the pretest Grade 9 
EXPLORE?” of student i. 


The results are summarized in Table 24, on the next page. The findings based on these results 
for the prediction of the posttest Grade 9 EXPLORE? scores are similar to those based on the 
results in Table 22 for the prediction of the posttest Grade 9 GMRT® scores for the same set of 
predictors, with the exception that the regression coefficient for special education in Table 24 is 
statistically significant (Bs = 0.778, p < .001). The positive sign of this regression coefficient 
indicates that the predicted posttest Grade 9 EXPLORE® score for special education students is 
less than one point (0.778) higher than that for non-special education students when controlling 
for all other predictors under the model in Equation 5. One may argue that, despite its statistical 
significance, the unique contribution of the special education variable to the prediction of the 
posttest Grade 9 EXPLORE? scores (in favor of special education students) is practically 
negligible. 
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Table 24. OSL regression for the prediction of posttest scores on the Grade 9 EXPLORE 


Variable B SE(B) p-value 
Constant 9.195 0.721 < .000 
Treatment (PRJ) -0.269 0.161 095 
Pretest 0.326 0.063 < .001 
Gender -0.199 0.164 226 
African-American -0.154 0.166 353 
Special education 0.778 0.213 < .001 
Free and reduced lunch -0.039 0.210 852 
Limited English proficiency 0.011 1.170 .992 

Additional Analyses 


Experimental analyses 


Subgroup analyses were also conducted to investigate for possible interactions between the 
treatment condition, PRJ III, and each of the five student-related variables used in this study: 
gender, ethnicity, special education, FARM, and LEP, while controlling for pretest scores on the 
Grade 9 GMRT®. Specifically, a two-factor analysis of covariance (ANCOVA) was performed 
five times, with the posttest scores on the grade 9 GMRT® as a dependent variable, the pretest 
scores on the grade 9 GMRT® as a covariate, and the factors being the treatment condition (1 = 
PRJ, 0 = Control) and each the five student-related variables, respectively. In all five ANCOVA 
analyses, the assumptions of normality, homogeneity of variances, and homogeneity of 
regression slopes were met. The only statistically significant effect in all five ANCOVAs was 
related to the interaction between treatment condition and gender, F (1, 827) = 4.43, p = .005, 
py = 0.005. At the same time, however, the partial eta squared value (pn? = 0.005) indicates 
that the effect size for the interaction effect is negligible as only 0.5% of the variance in the 
dependent variable is accounted for by the interaction effect between the two factors (PRJ and 
gender) while controlling for their main effects. With this in mind, the results showed that this 
interaction is of disordinal type, with the PRJ males scoring slightly higher than the control 
group males and, conversely, the PRJ females scoring slightly lower than the control group 
females. The descriptive statistics (ANCOVA adjusted means and standard deviations) are given 
in Table 25. Note that sample sizes reflect the analytical groups that contain imputed data. 


Table 25. Adjusted means and their standard deviations on the posttest Grade 9 GMRT® by 
treatment condition and gender 


Group n M SD 
Treatment (PRJ) 
Male 239 518.84 20.61 
Female 182 516.31 20.28 
Control 
Male 238 516.05 19.68 
Female 181 517.32 22.78 
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The five ANCOVA analyses were conducted again but, instead of the posttest Grade 9 GMRT® 
scores, this time the dependent variable was the posttest Grade 9 EXPLORE” scores. The only 
statistically significant effect now happens to be the interaction between treatment condition and 
ethnicity (1 = African-America, 0 = Other), F (1, 827) =5.03, p = .025, py’ = 0.006. The 
descriptive statistics (ANCOVA adjusted means and standard deviations) are given in Table 26. 
Clearly, despite the statistical significance of this interaction, the difference between the adjusted 
posttest Grade 9 EXPLORE® scores for African-American and other students are practically 
negligible. 


Table 26. Adjusted means and standard deviations on the posttest Grade 9 EXPLORE® by 
treatment condition and ethnicity’ 


Group n M SD 
Treatment (PRJ) 
African American 243 12.09 0.15 
Other 178 11.86 0.18 
Control 
African American 243 12.05 0.15 
Other 176 12.54 0.18 


Pn 27 a 7 
African American = 1; Other = 0 


The overall finding from the ANCOVA analyses, in consistency with the regression analyses for 
the same variables on the same data, is that there are no main effects for treatment condition and 
student-related variables (gender, ethnicity, special education, FARM, and LEP) for any of the 
two dependent variables used in this study—the posttest Grade 9 GMRT® scores and the posttest 
Grade 9 EXPLORE® scores of the students. There are no interactions between the treatment 
condition and the student-related variables, with the exception of negligible interaction effects 
between treatment condition and gender for the posttest Grade 9 GMRT® scores and between 
treatment condition and ethnicity (1 = African-American, 0 = Other) for the posttest Grade 9 
EXPLORE® scores as the dependent variable. 
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CONCLUSION 


Illinois Striving Readers was a complex project that required schools to implement a randomized 
assignment of adolescent struggling reader students to treatment and control classrooms. It also 
required teachers to learn and implement a new reading program to some but not all of those 
struggling students while being observed. Randomization is not a common procedure in public 
schools, nor is learning a new program as part of a rigorous study that involves so much scrutiny. 
Despite these challenges, the project had a successful first year, with the classroom 
implementation attaining middle to high levels of implementation in five of the participating 
schools, and the randomization process maintaining its integrity in all six schools. 


The planning year USED incorporated into the grant should be highlighted as a major contributor 
for this success. From the perspective of the implementation, the planning year provided time for 
the representatives from the state, local education agencies, and schools to familiarize themselves 
with each other, ask questions and solve concerns. The developers had time to address the 
questions brought by the school personnel before the actual instruction started, and propose 
adaptations to the intervention to address the uniqueness of each school. At the start of the 
implementation year, all participants had a clear idea of their roles, responsibilities, and 
expectations. They also had become familiar with the intervention, and when challenges started 
to appear as the implementation progressed, implementers and developers were able to work 
closely together to address the challenges. 


From the perspective of the study, the planning year was essential for the evaluators to build trust 
with the implementers, learn about the different policies and regulations that could affect the 
study in each of the participant school districts, while addressing questions and assuaging fears 
from school administrators and school division representatives regarding the random assignment. 
Furthermore, the planning year provided the evaluators with enough time to become familiar 
with the intervention and to develop and test data collection instruments that better reflected the 
intervention model. 


In April 2010, Striving Readers grantees and evaluators were notified that funds for the Striving 
Readers program had been cut by Congress and the project was to end as the first 
implementation year came to a close. In talking to the project participants, the announcement 
appeared to have caused no clear disruption in the implementation process. Four of the six RITs 
had tenure at their school districts and knew they would be assured a job, while the two recently 
hired RITs were also confident they would find jobs. The main comments regarding the process 
was the lack of time for the RITs to learn the intervention and work out potential challenges 
before the impact study was completed. As one VIS observed, “The first year there are always 
kinks. Next year will be better because they’ll know the procedures and will know about how to 
get started.” Likewise, one of the RITs commented, 


The first year of a new curriculum . . . is always challenging. After we all got used to the 
routine, the students began to move forward... I am disappointed that the grant has 
ended, but I am thankful that my administration feels [PRJ] is an important part of the 
high school and decided to keep me here. 
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At least two of the school districts decided to continue the intervention after the grant expired. 
The RITs in these two school districts expressed feeling happier that they would continue with 
the intervention without the constraints of the study. According to a RIT in one of these districts, 


I will be having this same position for the 2011-2012 school year. We will select students 
based on need and attendance. A student will stay in the program only if their behavior 
allows others in the class to learn. I will be doing a small group rotation for the 
structure of the program. This allows students more of a variety throughout the lesson 
and will give me a chance to work with small groups of students on or close to their 
[reading] levels. I will still introduce the story as whole group but then move into small 
groups. I look forward to next years’ experience. 


From the study’s perspective, the sudden reduction in funds led the evaluators to curtail the 
evaluation activities related to implementation in order to conserve resources for the impact 
study and reporting, following USED recommendations. Plans to collect attendance data or 
further explore some of the preliminary findings had to be cancelled and the implementation 
study came to an abrupt halt. The school personnel also mentioned that after the termination of 
the Striving Readers funds, school staff was reluctant to pursue testing with the control group 
students, particularly with a testing instrument that was used specifically as an evaluation tool for 
the grant. This reluctance was reflected in the number of control group students without testing 
data. 


Two lessons can be learned from this “rise and fall” of the second cohort of Striving Readers 
studies. The first lesson is that, for any study requiring schools to go outside their routines and 
incorporate randomized controlled trials, a planning year will certainly facilitate the 
understanding of the process, the buy-in, and therefore, the integrity of the randomization. The 
planning year also allows for building trust among the different players, including implementers, 
program developers and evaluators. This trust is essential to ensure open channels of 
communication, particularly when obstacles arise during the implementation. The second lesson 
is that a rigorous study should be given time to be completed in order to avoid the risk of 
presenting only provisional results that must be taken with caution. More importantly, such an 
abrupt end of a study that required so much initial commitment and preparation from 
implementers also risk leaving participants afraid of engaging in similar projects in the future. In 
the end, education research suffers. 
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APPENDIX A: Summary Statistics for Outcome Measures 


Unadjusted means and standard deviations on the posttest Grade 9 EXPLORE® by treatment 


condition 

Mean SD n 
Treatment 11.99 2.824 264 
Control 12.34 2.889 250 


Unadjusted means and standard deviations on the posttest Grade 9 GMRT by treatment 


condition 

Mean SD n 
Treatment 517.10 23.650 232 
Control 517.16 25.824 216 


*Note that the n’s in this table add up to 448, which is 12 fewer than the final analytical sample presented in the 
consort diagram (Figure 4). This is due to 12 students — 6 in the treatment and 6 in control — having post-test data 
for only one component of the GMRT, and thus having no total GMRT score 
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APPENDIX B: Implementation study protocols 


Illinois Striving Readers 
Classroom Observation Rubric 


Instructions to observer 


1. 


Observers should stay for the whole class; if you need to leave the room before the end of 
the class, please check here and explain the reason in the back of this page 


Before starting the observation, 


a. Ask the teacher the number of the Expedition you will observe (from 1 to 10 in 
the Expeditions sequence). 

b. Within the Expedition, ask the teacher the number of the word study lesson she is 
teaching that day. 


. If you answer partially or no to any item in your observation forms, please use the back 


of the page to explain your answer. 


. In the classroom behavior/management table, if the behavior is not applicable during part 


of the observation time, write n/a and deduct that period from the score (e.g. instead of 
dividing by 5, divide by 4 or what is applicable). 


Use back of the observation page to enter comments. You don’t need to comment in 
every aspect of your observation but make short comments about behaviors or events that 
catch your attention and can be relevant to a better understanding of why the lesson 
occurred the way it did. 


Note 


Illinois has two days for lesson 5 and two days for lesson 10. Each expedition takes a 
total of 12 days 


Illinois Striving Readers, Implementation Year One Report 56 


OVERVIEW 


School: Grade_ Teacher Name: 

Observer: Observation Date: __ Observation Time: 
Lesson Number: Expedition Number: ___————SsSNNo.. of students: 

Was the entire lesson completed in the class period? Yes__—No__ 

In addition to the teacher, is there another adult in the room? Yes _ No 
Who? (circle) Special education teacher Special education aide Voyager coach 
School administrator School district staff Illinois Project Director Other 


A. Classroom Environment 


Yes Partially No 
(complete at beginning of lesson) 
1. Teachers have sufficient space to conduct individual and/or 
group work 
2. Instructional areas are clearly identified (i.e. whole group, 
independent small group, word study) 
3. Teacher resources for the daily lesson are readily available 
4. All students have readily available materials, as needed 
B. Lesson Planning and Delivery — overview : Not 
Yes Partially No 
(complete at the end of lesson) observed 


5. Teacher closely follows the curriculum guide during 
instruction 


6. Pace is brisk and business-like, yet personal 


7. Skills are modeled correctly 


The steps of the correction procedures are followed as 
needed 


9. Teacher puts students into groups as indicated by the 
lesson 


10. Teacher uses built-in differentiated instruction 
strategies as needed : 

[| re-teach lesson 

word study lesson 

English Language Learner strategies 

challenge questions 

Paired reading 


Illinois Striving Readers, Implementation Year One Report 


57 


Lesson 1 Form 


C. Lesson Planning and Delivery-lesson specific 


(Check the box next to each activity you observe ) 


Components 
delivered in 
order? 


(Y/N) 


Steps 
delivered 
in order? 

(Y/N) 


Components 
delivered within 
allotted time? 
(Y/N) 


WHOLE GROUP 

Introduce the Expedition (10-15 min.) 
Discuss probing questions 
Start time End time 


Before Reading (15 min.) 
Introduce vocabulary 
Start time End time 
Introduce the target skill 
Start time End time 
Introduce the passage 
Start time End time 


During Reading (10-15 min.) 
[1 Students read text 
Start time End time 


After Reading (5-10 min.) 
Check comprehension 
Start time End time 


INDEPENDENT 
[| Students’ practice vocabulary using the 
online technology component 

Start time End time 

Students’ select books for independent 
reading 

Start time 


End time 


D. Classroom Behavior/Management 


Minutes 


Total 


10 10 10 


10 10 


teacher or following teacher instructions 


Half or more of the students are paying attention to 


Half or more of the students are responding to 
teacher questions or prompts 


Half or more of the students are actively 


participating in the activities assigned by the teacher 


(group or individually) 


Teacher addresses student behavior promptly to 
minimize disruption in the classroom 


Teacher makes an effort to involve students who 
appear disengaged 


Students follow expectations for working in groups 


D. Classroom Behavior/Management Scale: 
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WHOLE GROUP 
Prepare to Reread (15 min.) 


Introduce the lesson 


Lesson 2/Lesson 4/Lesson 7/Lesson 9 Form 


Start time. End time 

Practice vocabulary 

Start time. End time 
Reread (25-30 min.) 

Apply the target skill 

Start time____—_ End time 

Write in response to reading 

Start time____— End time 


immediately. 


Expedition Organizer (5-10 min.) 
This is meant to be a quick activity. There shouldn’t be a lot of 
thought needed to fill in Parts A & B. It is spread out over 
several lessons, so all parts do not need to be finished 


Build Background DVD (5-10 min.) Lesson 9 ONLY 


While the larger group of students is independent reading, a small group of students may be involved in 
teacher-directed Word Study. 


1. Half or more of the students are paying attention to 
teacher or following teacher instructions 


2. Half or more of the students are responding to 
teacher questions or prompts 


3. Half or more of the students are actively 
participating in the activities assigned by the 
teacher (group or individually) 


4. Teacher addresses student behavior promptly to 
minimize disruption in the classroom 


appear disengaged 


5. Teacher makes an effort to involve students who 


6. Students follow expectations for working in groups 


D. Classroom Behavior/Management Scale: 1 Not At All 2 Occasionally 
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Lesson 3/Lesson 6/ Lesson 8 Form 


WHOLE GROUP 
Before Reading (15-20 min.) 


Introduce vocabulary 

Start time End time 
Introduce the target skill 

Start time. End time___ 
Introduce the passage 

Start time. End time__ 


During Reading (20-25 min.) 
Students read text 
Start time End time 


After Reading (5-10 min.) 
Check comprehension 
Start time End time 


INDEPENDENT 
Students practice vocabulary using the 
online technology component 


Start time End time 

Students select books for independent 
reading 

Start time End time 


1. Half or more of the students are paying attention 
to teacher or following teacher instructions 

2. Half or more of the students are responding to 
teacher questions or prompts 

3. Half or more of the students are actively 
participating in the activities assigned by the 
teacher (group or individually) 

4. Teacher addresses student behavior promptly to 
minimize disruption in the classroom 

5. Teacher makes an effort to involve students who 
appear disengaged 

6. Students follow expectations for working in 
groups 

D. Classroom Behavior/Management Scale: 1 Not At All 2 Occasionally 3 Frequently 
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Lesson 5/10 Forms 


WHOLE GROUP - teacher selects any or all options 
Review, Extend, Assess 


Review vocabulary 

Extend vocabulary 

Assess Comprehension and Vocabulary 
Reteach 

Passport Reading Journeys Library 
Technology 

Expedition Project 

Writing Process 


While the larger group of students is independent reading, a small group of students may be involved in 
teacher-directed Word Study. 


1. Half or more of the students are paying attention to 
teacher or following teacher instructions 

2. Half or more of the students are responding to 
teacher questions or prompts 

3. Half or more of the students are actively 
participating in the activities assigned by the teacher 
(group or individually) 

4. Teacher addresses student behavior promptly to 
minimize disruption in the classroom 

5. Teacher makes an effort to involve students who 
appear disengaged 


6. Students follow expectations for working in groups 


D. Classroom Behavior/Management Scale: 1 Not At All 2 Occasionally 3 Frequently 
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ILLINOIS STRIVING READERS 
READING SPECIALIST CHECK-IN PROTOCOL 


School 
(Check the school where the interventionist you are calling works) 
Springfield Decatur Kankakee 
Lanphier Eisenhower 
Springfield Southeast MacArthur Douglas Danville 
Date of Call 
(Please write the day in the corresponding month of the call) 
2010 2011 
September January 
October February 
November March 
December April 
May 


Note: Long interview with interventionists in June 


Professional Development 


1. What professional development (PD) have you attended this month? 


None 

Voyager-related PD for _ hours topic: 
School-based PD for hours topic: 
Other: number of hours___—_—stoppic: 


2. If the answer to question 1 is no, why? 


No PD was offered this month 


Lack of time 


Lack of interest 


Other reasons (specify)_ 
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Coaching/Mentoring Support 


3. Did you receive coaching/mentoring support this month from 


The Voyager Implementation Specialist 


School staff (assistant principal, other) 


Illinois Board of Education 


Other reasons (specify)_ 


4. How many hours of coaching/mentoring support did you receive this month? 
5. What was the focus of the meeting? 


6. If you did not receive any coaching or mentoring support this month, why not? 


I will be receiving support next month 


I don’t know if I will be receiving coaching or mentoring support 


I have been too busy 


Other (specify) 


7. How would you rate the coaching or mentoring support you received this month? 


Not relevant Very relevant 
1 2 3 4 5 
Insufficient Sufficient 
1 2 3 4 5 


8. How aligned is the training and support you received from the coach, the project director, 


the liaison, and any other Voyager staff? 


Conflicting Very aligned 
1 2 3 4 5 
Instruction 


9. How was attendance in your classes this month? 


Almost all students attended 


____ Number of students chronically absent (more than once a week) 
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____ Students were suspended (OSS) for ___ days 
____ Students were suspended (ISS) for ___ days 
The school was closed for____——— days 
Ihad___ class days cancelled due to 


10. What Expedition and Lesson are you on in each of your classes? 


Expedition lesson 


11. In relation to the pacing guide, how far are you from where you should be at this time of 


the year? 
Iam ahead by___ lessons in_____ classes (number of classes) 
Iam behind by___ lessons in___ classes 
Iamonschedulein _ classes 


12. What activities did you choose for the 5" and 10" days of the Expedition? 


13. Why did you make these choices? 


Use of time 
14. Can you say that at least 80% of your time this month has been dedicated to teaching 
Voyager? 
Yes No 
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15. If not, what other duties have you been asked to do? 


16. Do you have scheduled time to plan your classes during the school day? 


Yes No 
Conclusion 


17. Is there anything you would like to share with us at this time about the implementation of 


the program? 
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Interview Protocol - coaches 
Date 
Site 


1. How frequently do you meet with the teachers? 


2. What is the main focus of her work with the teachers? 


3. What issues the teachers are encountering? 


4. How much support are they receiving from principals? 


5. What factors do you feel influence the implementation of the intervention? 


6. Other comments? 
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